2024-09-16T05:11:02.637 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T05:11:02.650 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T05:11:02.717 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906136 branch: squid description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/misc} email: ceph-qa@ceph.com first_in_suite: false job_id: '7906136' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 8921c8ef947807b717db041d85f0761b7914d5fc ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 8921c8ef947807b717db041d85f0761b7914d5fc selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 workunit: branch: squid sha1: ad35db06e8924912f01e32b35576297c0f1352b4 owner: scheduled_teuthology@teuthology priority: 700 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 5616 sha1: 8921c8ef947807b717db041d85f0761b7914d5fc sleep_before_teardown: 0 subset: 15927/100000 suite: rados suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: ad35db06e8924912f01e32b35576297c0f1352b4 targets: smithi178.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDwYpaen2yn7s+h0Wp7ZzKxybK42lLyisZOjKQ8m2DLgzvAfX6X6ZMgahcoZhGJKOyM9wUnGnkRPnyM+rwfD9Lc= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - misc teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_21:00:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T05:11:02.717 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa; will attempt to use it 2024-09-16T05:11:02.718 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks 2024-09-16T05:11:02.718 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T05:11:02.720 INFO:teuthology.task.internal:Checking packages... 2024-09-16T05:11:02.741 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '8921c8ef947807b717db041d85f0761b7914d5fc' 2024-09-16T05:11:02.741 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T05:11:02.741 INFO:teuthology.packaging:ref: None 2024-09-16T05:11:02.741 INFO:teuthology.packaging:tag: None 2024-09-16T05:11:02.741 INFO:teuthology.packaging:branch: squid 2024-09-16T05:11:02.741 INFO:teuthology.packaging:sha1: 8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T05:11:02.741 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-16T05:11:02.966 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-328.gad35db06 2024-09-16T05:11:02.967 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T05:11:02.973 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T05:11:02.973 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T05:11:03.012 INFO:teuthology.task.internal:Saving configuration 2024-09-16T05:11:03.020 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T05:11:03.072 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T05:11:03.093 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi178.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906136', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 05:05:30.267432', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDwYpaen2yn7s+h0Wp7ZzKxybK42lLyisZOjKQ8m2DLgzvAfX6X6ZMgahcoZhGJKOyM9wUnGnkRPnyM+rwfD9Lc='} 2024-09-16T05:11:03.094 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T05:11:03.099 INFO:teuthology.task.internal:roles: ubuntu@smithi178.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-16T05:11:03.099 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T05:11:03.144 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe967dd3ac0>, signals=[15]) 2024-09-16T05:11:03.145 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T05:11:03.150 INFO:teuthology.task.internal:Opening connections... 2024-09-16T05:11:03.150 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi178.front.sepia.ceph.com 2024-09-16T05:11:03.152 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T05:11:03.229 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T05:11:03.235 DEBUG:teuthology.orchestra.run.smithi178:> uname -m 2024-09-16T05:11:03.311 INFO:teuthology.orchestra.run.smithi178.stdout:x86_64 2024-09-16T05:11:03.312 DEBUG:teuthology.orchestra.run.smithi178:> cat /etc/os-release 2024-09-16T05:11:03.370 INFO:teuthology.orchestra.run.smithi178.stdout:NAME="CentOS Stream" 2024-09-16T05:11:03.371 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION="9" 2024-09-16T05:11:03.371 INFO:teuthology.orchestra.run.smithi178.stdout:ID="centos" 2024-09-16T05:11:03.371 INFO:teuthology.orchestra.run.smithi178.stdout:ID_LIKE="rhel fedora" 2024-09-16T05:11:03.371 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION_ID="9" 2024-09-16T05:11:03.371 INFO:teuthology.orchestra.run.smithi178.stdout:PLATFORM_ID="platform:el9" 2024-09-16T05:11:03.371 INFO:teuthology.orchestra.run.smithi178.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T05:11:03.371 INFO:teuthology.orchestra.run.smithi178.stdout:ANSI_COLOR="0;31" 2024-09-16T05:11:03.371 INFO:teuthology.orchestra.run.smithi178.stdout:LOGO="fedora-logo-icon" 2024-09-16T05:11:03.371 INFO:teuthology.orchestra.run.smithi178.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T05:11:03.371 INFO:teuthology.orchestra.run.smithi178.stdout:HOME_URL="https://centos.org/" 2024-09-16T05:11:03.371 INFO:teuthology.orchestra.run.smithi178.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T05:11:03.371 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T05:11:03.371 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T05:11:03.372 INFO:teuthology.lock.ops:Updating smithi178.front.sepia.ceph.com on lock server 2024-09-16T05:11:03.389 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T05:11:03.398 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T05:11:03.428 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T05:11:03.429 DEBUG:teuthology.orchestra.run.smithi178:> test '!' -e /home/ubuntu/cephtest 2024-09-16T05:11:03.446 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T05:11:03.453 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T05:11:03.453 DEBUG:teuthology.orchestra.run.smithi178:> test -z $(ls -A /var/lib/ceph) 2024-09-16T05:11:03.508 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T05:11:03.535 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T05:11:03.548 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-16T05:11:03.549 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-16T05:11:03.549 DEBUG:teuthology.orchestra.run.smithi178:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T05:11:03.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:11:03.566 DEBUG:teuthology.orchestra.run.smithi178:> uname -r 2024-09-16T05:11:03.623 INFO:teuthology.orchestra.run.smithi178.stdout:5.14.0-503.el9.x86_64 2024-09-16T05:11:03.623 INFO:teuthology.task.kernel:Running kernel on smithi178: 5.14.0-503.el9.x86_64 2024-09-16T05:11:03.624 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum install -y kernel 2024-09-16T05:11:07.104 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 8.2 MB 00:01 2024-09-16T05:11:08.888 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - AppStream 30 MB/s | 20 MB 00:00 2024-09-16T05:11:14.771 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - CRB 3.1 MB/s | 6.5 MB 00:02 2024-09-16T05:11:16.339 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - Extras packages 79 kB/s | 19 kB 00:00 2024-09-16T05:11:17.351 INFO:teuthology.orchestra.run.smithi178.stdout:Extra Packages for Enterprise Linux 25 MB/s | 23 MB 00:00 2024-09-16T05:11:22.798 INFO:teuthology.orchestra.run.smithi178.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-09-16T05:11:24.813 INFO:teuthology.orchestra.run.smithi178.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T05:11:24.813 INFO:teuthology.orchestra.run.smithi178.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T05:11:24.916 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-09-16T05:11:24.927 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:11:24.927 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2024-09-16T05:11:24.927 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:11:24.927 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-09-16T05:11:24.927 INFO:teuthology.orchestra.run.smithi178.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T05:11:24.927 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-09-16T05:11:24.927 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T05:11:24.927 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T05:11:24.927 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T05:11:24.927 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:11:24.927 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-09-16T05:11:24.928 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:11:24.928 INFO:teuthology.orchestra.run.smithi178.stdout:Install 4 Packages 2024-09-16T05:11:24.928 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:11:24.928 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 79 M 2024-09-16T05:11:24.928 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 126 M 2024-09-16T05:11:24.928 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-09-16T05:11:25.066 INFO:teuthology.orchestra.run.smithi178.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 262 kB/s | 23 kB 00:00 2024-09-16T05:11:25.408 INFO:teuthology.orchestra.run.smithi178.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 36 MB/s | 16 MB 00:00 2024-09-16T05:11:25.758 INFO:teuthology.orchestra.run.smithi178.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 41 MB/s | 29 MB 00:00 2024-09-16T05:11:26.150 INFO:teuthology.orchestra.run.smithi178.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 29 MB/s | 35 MB 00:01 2024-09-16T05:11:26.150 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:11:26.150 INFO:teuthology.orchestra.run.smithi178.stdout:Total 64 MB/s | 79 MB 00:01 2024-09-16T05:11:26.829 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-09-16T05:11:27.202 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-09-16T05:11:27.202 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-09-16T05:11:28.144 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-09-16T05:11:28.144 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-09-16T05:11:30.582 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-09-16T05:11:31.299 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T05:11:31.375 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T05:11:34.175 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T05:11:34.298 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T05:11:39.166 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T05:11:39.206 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T05:11:44.039 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T05:12:22.047 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T05:12:22.086 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T05:12:22.392 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T05:12:22.392 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T05:12:22.392 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T05:12:22.392 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T05:12:23.211 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T05:12:23.212 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:12:23.212 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-09-16T05:12:23.212 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T05:12:23.212 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T05:12:23.212 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T05:12:23.212 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T05:12:23.212 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:12:23.212 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-09-16T05:12:23.546 DEBUG:teuthology.orchestra.run.smithi178:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T05:12:23.638 INFO:teuthology.orchestra.run.smithi178.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T05:12:23.639 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T05:12:23.639 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T05:12:23.639 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T05:12:23.639 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T05:12:23.639 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-16T05:12:23.639 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi178.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T05:12:23.639 DEBUG:teuthology.orchestra.run.smithi178:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T05:12:23.717 INFO:teuthology.orchestra.run.smithi178.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T05:12:23.718 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T05:12:23.718 DEBUG:teuthology.orchestra.run.smithi178:> sudo rpm -qi grub2-tools 2024-09-16T05:12:23.757 INFO:teuthology.orchestra.run.smithi178.stdout:Name : grub2-tools 2024-09-16T05:12:23.757 INFO:teuthology.orchestra.run.smithi178.stdout:Epoch : 1 2024-09-16T05:12:23.757 INFO:teuthology.orchestra.run.smithi178.stdout:Version : 2.06 2024-09-16T05:12:23.757 INFO:teuthology.orchestra.run.smithi178.stdout:Release : 82.el9 2024-09-16T05:12:23.757 INFO:teuthology.orchestra.run.smithi178.stdout:Architecture: x86_64 2024-09-16T05:12:23.757 INFO:teuthology.orchestra.run.smithi178.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T05:12:23.757 INFO:teuthology.orchestra.run.smithi178.stdout:Group : Unspecified 2024-09-16T05:12:23.758 INFO:teuthology.orchestra.run.smithi178.stdout:Size : 8274670 2024-09-16T05:12:23.758 INFO:teuthology.orchestra.run.smithi178.stdout:License : GPLv3+ 2024-09-16T05:12:23.758 INFO:teuthology.orchestra.run.smithi178.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T05:12:23.758 INFO:teuthology.orchestra.run.smithi178.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T05:12:23.758 INFO:teuthology.orchestra.run.smithi178.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T05:12:23.758 INFO:teuthology.orchestra.run.smithi178.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T05:12:23.758 INFO:teuthology.orchestra.run.smithi178.stdout:Packager : builder@centos.org 2024-09-16T05:12:23.758 INFO:teuthology.orchestra.run.smithi178.stdout:Vendor : CentOS 2024-09-16T05:12:23.758 INFO:teuthology.orchestra.run.smithi178.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T05:12:23.758 INFO:teuthology.orchestra.run.smithi178.stdout:Summary : Support tools for GRUB. 2024-09-16T05:12:23.758 INFO:teuthology.orchestra.run.smithi178.stdout:Description : 2024-09-16T05:12:23.758 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:12:23.758 INFO:teuthology.orchestra.run.smithi178.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T05:12:23.759 INFO:teuthology.orchestra.run.smithi178.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T05:12:23.759 INFO:teuthology.orchestra.run.smithi178.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T05:12:23.759 INFO:teuthology.orchestra.run.smithi178.stdout:hardware devices. 2024-09-16T05:12:23.759 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:12:23.759 INFO:teuthology.orchestra.run.smithi178.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T05:12:23.760 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T05:12:23.760 INFO:teuthology.task.kernel:Updating grub on smithi178 to boot 5.14.0-505.el9.x86_64 2024-09-16T05:12:23.760 DEBUG:teuthology.orchestra.run.smithi178:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T05:12:24.391 INFO:teuthology.orchestra.run.smithi178.stderr:Generating grub configuration file ... 2024-09-16T05:12:25.799 INFO:teuthology.orchestra.run.smithi178.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T05:12:25.822 INFO:teuthology.orchestra.run.smithi178.stderr:done 2024-09-16T05:12:25.825 DEBUG:teuthology.orchestra.run.smithi178:> mktemp 2024-09-16T05:12:25.841 INFO:teuthology.orchestra.run.smithi178.stdout:/tmp/tmp.TViZKq9FVR 2024-09-16T05:12:25.841 DEBUG:teuthology.orchestra.run.smithi178:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.TViZKq9FVR 2024-09-16T05:12:25.907 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 0666 /tmp/tmp.TViZKq9FVR 2024-09-16T05:12:26.060 DEBUG:teuthology.orchestra.remote:smithi178:/tmp/tmp.TViZKq9FVR is 6KB 2024-09-16T05:12:26.109 DEBUG:teuthology.orchestra.run.smithi178:> rm -fr /tmp/tmp.TViZKq9FVR 2024-09-16T05:12:26.124 DEBUG:teuthology.orchestra.run.smithi178:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T05:12:26.191 INFO:teuthology.orchestra.run.smithi178.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T05:12:26.191 INFO:teuthology.orchestra.run.smithi178.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T05:12:26.191 INFO:teuthology.orchestra.run.smithi178.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T05:12:26.191 INFO:teuthology.orchestra.run.smithi178.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T05:12:26.192 DEBUG:teuthology.orchestra.run.smithi178:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T05:12:26.350 DEBUG:teuthology.orchestra.run.smithi178:> sudo shutdown -r now 2024-09-16T05:12:56.379 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-16T05:12:56.380 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-09-16T05:12:56.381 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T05:13:30.941 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.178 2024-09-16T05:13:39.949 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-09-16T05:13:39.950 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T05:13:43.005 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.178 2024-09-16T05:13:55.017 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-09-16T05:13:55.018 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T05:14:27.090 DEBUG:teuthology.orchestra.run.smithi178:> true 2024-09-16T05:14:27.458 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-09-16T05:14:27.458 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-16T05:14:27.458 DEBUG:teuthology.orchestra.run.smithi178:> uname -r 2024-09-16T05:14:27.473 INFO:teuthology.orchestra.run.smithi178.stdout:5.14.0-505.el9.x86_64 2024-09-16T05:14:27.473 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T05:14:27.473 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T05:14:27.473 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T05:14:28.474 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-16T05:14:28.474 DEBUG:teuthology.orchestra.run.smithi178:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T05:14:28.561 INFO:teuthology.orchestra.run.smithi178.stdout:ttyS1 2024-09-16T05:14:28.578 DEBUG:teuthology.parallel:result is None 2024-09-16T05:14:28.579 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T05:14:28.588 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T05:14:28.588 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T05:14:28.606 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T05:14:28.614 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T05:14:28.649 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T05:14:28.649 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T05:14:28.698 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T05:14:28.705 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T05:14:28.706 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T05:14:28.744 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:14:28.756 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:14:28.757 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T05:14:28.765 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T05:14:28.765 DEBUG:teuthology.orchestra.run.smithi178:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T05:14:28.825 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T05:14:28.835 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T05:14:28.835 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T05:14:28.882 DEBUG:teuthology.orchestra.run.smithi178:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T05:14:28.967 DEBUG:teuthology.orchestra.run.smithi178:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T05:14:29.012 DEBUG:teuthology.orchestra.run.smithi178:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T05:14:29.095 DEBUG:teuthology.orchestra.run.smithi178:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T05:14:29.136 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-09-16T05:14:29.137 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T05:14:29.201 DEBUG:teuthology.orchestra.run.smithi178:> sudo service rsyslog restart 2024-09-16T05:14:29.283 INFO:teuthology.orchestra.run.smithi178.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T05:14:29.558 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T05:14:29.566 INFO:teuthology.task.internal:Starting timer... 2024-09-16T05:14:29.566 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T05:14:29.611 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T05:14:29.644 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-09-16T05:14:29.644 DEBUG:teuthology.orchestra.run.smithi178:> sudo service auditd rotate 2024-09-16T05:14:29.706 INFO:teuthology.orchestra.run.smithi178.stdout:Rotating logs: 2024-09-16T05:14:29.706 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T05:14:29.706 DEBUG:teuthology.orchestra.run.smithi178:> /usr/sbin/getenforce 2024-09-16T05:14:29.734 INFO:teuthology.orchestra.run.smithi178.stdout:Permissive 2024-09-16T05:14:29.734 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi178.front.sepia.ceph.com': 'permissive'} 2024-09-16T05:14:29.734 DEBUG:teuthology.orchestra.run.smithi178:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T05:14:29.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:14:29.799 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T05:14:29.799 DEBUG:teuthology.orchestra.run.smithi178:> sudo /usr/sbin/setenforce permissive 2024-09-16T05:14:29.872 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T05:14:29.882 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T05:14:29.895 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T05:14:29.896 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi178.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T05:18:12.379 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi178.front.sepia.ceph.com')] 2024-09-16T05:18:12.380 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-09-16T05:18:12.382 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T05:18:12.460 DEBUG:teuthology.orchestra.run.smithi178:> true 2024-09-16T05:18:12.535 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-09-16T05:18:12.535 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T05:18:12.549 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T05:18:12.549 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T05:18:12.550 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T05:18:12.603 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T05:18:12.618 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T05:18:12.642 INFO:teuthology.orchestra.run.smithi178.stderr:sudo: ntpd: command not found 2024-09-16T05:18:12.653 INFO:teuthology.orchestra.run.smithi178.stdout:506 Cannot talk to daemon 2024-09-16T05:18:12.667 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T05:18:12.681 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T05:18:12.750 INFO:teuthology.orchestra.run.smithi178.stderr:bash: line 1: ntpq: command not found 2024-09-16T05:18:12.754 INFO:teuthology.orchestra.run.smithi178.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T05:18:12.754 INFO:teuthology.orchestra.run.smithi178.stdout:=============================================================================== 2024-09-16T05:18:12.754 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T05:18:12.754 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T05:18:12.754 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T05:18:12.754 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T05:18:12.755 INFO:teuthology.run_tasks:Running task install... 2024-09-16T05:18:12.765 DEBUG:teuthology.task.install:project ceph 2024-09-16T05:18:12.765 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}} 2024-09-16T05:18:12.765 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'} 2024-09-16T05:18:12.766 INFO:teuthology.task.install:Using flavor: default 2024-09-16T05:18:12.771 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T05:18:12.772 INFO:teuthology.task.install:extra packages: [] 2024-09-16T05:18:12.772 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'tag': None, 'wait_for_package': False} 2024-09-16T05:18:12.772 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T05:18:12.921 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/ 2024-09-16T05:18:12.921 INFO:teuthology.task.install.rpm:Package version is 19.1.1-326.g8921c8ef 2024-09-16T05:18:13.030 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T05:18:13.030 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-09-16T05:18:13.030 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T05:18:13.057 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-16T05:18:13.057 DEBUG:teuthology.orchestra.run.smithi178:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8921c8ef947807b717db041d85f0761b7914d5fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T05:18:13.122 DEBUG:teuthology.orchestra.run.smithi178:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T05:18:13.198 DEBUG:teuthology.orchestra.run.smithi178:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T05:18:13.262 INFO:teuthology.orchestra.run.smithi178.stdout:check_obsoletes = 1 2024-09-16T05:18:13.264 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum clean all 2024-09-16T05:18:13.495 INFO:teuthology.orchestra.run.smithi178.stdout:45 files removed 2024-09-16T05:18:13.521 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-radosgw 2024-09-16T05:18:14.059 INFO:teuthology.orchestra.run.smithi178.stdout:ceph packages for x86_64 325 kB/s | 83 kB 00:00 2024-09-16T05:18:14.335 INFO:teuthology.orchestra.run.smithi178.stdout:ceph noarch packages 50 kB/s | 12 kB 00:00 2024-09-16T05:18:14.586 INFO:teuthology.orchestra.run.smithi178.stdout:ceph source packages 8.8 kB/s | 1.9 kB 00:00 2024-09-16T05:18:15.827 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - BaseOS 6.8 MB/s | 8.2 MB 00:01 2024-09-16T05:18:17.828 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - AppStream 22 MB/s | 20 MB 00:00 2024-09-16T05:18:23.387 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.5 MB 00:01 2024-09-16T05:18:24.780 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - Extras packages 118 kB/s | 19 kB 00:00 2024-09-16T05:18:25.859 INFO:teuthology.orchestra.run.smithi178.stdout:Extra Packages for Enterprise Linux 24 MB/s | 23 MB 00:00 2024-09-16T05:18:31.331 INFO:teuthology.orchestra.run.smithi178.stdout:lab-extras 36 kB/s | 1.7 kB 00:00 2024-09-16T05:18:33.732 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-09-16T05:18:33.747 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:18:33.747 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-09-16T05:18:33.747 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:18:33.747 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-09-16T05:18:33.747 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-radosgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 11 M 2024-09-16T05:18:33.748 INFO:teuthology.orchestra.run.smithi178.stdout:Upgrading: 2024-09-16T05:18:33.748 INFO:teuthology.orchestra.run.smithi178.stdout: librados2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.4 M 2024-09-16T05:18:33.748 INFO:teuthology.orchestra.run.smithi178.stdout: librbd1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.2 M 2024-09-16T05:18:33.748 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-09-16T05:18:33.748 INFO:teuthology.orchestra.run.smithi178.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T05:18:33.748 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-base x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.4 M 2024-09-16T05:18:33.748 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 21 M 2024-09-16T05:18:33.748 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-selinux x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 25 k 2024-09-16T05:18:33.748 INFO:teuthology.orchestra.run.smithi178.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T05:18:33.748 INFO:teuthology.orchestra.run.smithi178.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T05:18:33.748 INFO:teuthology.orchestra.run.smithi178.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T05:18:33.748 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 751 k 2024-09-16T05:18:33.748 INFO:teuthology.orchestra.run.smithi178.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T05:18:33.749 INFO:teuthology.orchestra.run.smithi178.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T05:18:33.749 INFO:teuthology.orchestra.run.smithi178.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T05:18:33.749 INFO:teuthology.orchestra.run.smithi178.stdout: libradosstriper1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 506 k 2024-09-16T05:18:33.749 INFO:teuthology.orchestra.run.smithi178.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T05:18:33.749 INFO:teuthology.orchestra.run.smithi178.stdout: librgw2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.3 M 2024-09-16T05:18:33.749 INFO:teuthology.orchestra.run.smithi178.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T05:18:33.749 INFO:teuthology.orchestra.run.smithi178.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T05:18:33.749 INFO:teuthology.orchestra.run.smithi178.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T05:18:33.749 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-argparse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 45 k 2024-09-16T05:18:33.749 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 136 k 2024-09-16T05:18:33.749 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cephfs x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 161 k 2024-09-16T05:18:33.749 INFO:teuthology.orchestra.run.smithi178.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T05:18:33.749 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rados x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 320 k 2024-09-16T05:18:33.750 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 300 k 2024-09-16T05:18:33.750 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 99 k 2024-09-16T05:18:33.750 INFO:teuthology.orchestra.run.smithi178.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T05:18:33.750 INFO:teuthology.orchestra.run.smithi178.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T05:18:33.750 INFO:teuthology.orchestra.run.smithi178.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T05:18:33.750 INFO:teuthology.orchestra.run.smithi178.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T05:18:33.750 INFO:teuthology.orchestra.run.smithi178.stdout:Installing weak dependencies: 2024-09-16T05:18:33.750 INFO:teuthology.orchestra.run.smithi178.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T05:18:33.750 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:18:33.750 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-09-16T05:18:33.750 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:18:33.750 INFO:teuthology.orchestra.run.smithi178.stdout:Install 30 Packages 2024-09-16T05:18:33.750 INFO:teuthology.orchestra.run.smithi178.stdout:Upgrade 2 Packages 2024-09-16T05:18:33.751 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:18:33.751 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 60 M 2024-09-16T05:18:33.751 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-09-16T05:18:34.715 INFO:teuthology.orchestra.run.smithi178.stdout:(1/32): ceph-base-19.1.1-326.g8921c8ef.el9.x86_ 9.6 MB/s | 5.4 MB 00:00 2024-09-16T05:18:34.766 INFO:teuthology.orchestra.run.smithi178.stdout:(2/32): ceph-selinux-19.1.1-326.g8921c8ef.el9.x 506 kB/s | 25 kB 00:00 2024-09-16T05:18:34.866 INFO:teuthology.orchestra.run.smithi178.stdout:(3/32): libcephfs2-19.1.1-326.g8921c8ef.el9.x86 7.4 MB/s | 751 kB 00:00 2024-09-16T05:18:34.950 INFO:teuthology.orchestra.run.smithi178.stdout:(4/32): libradosstriper1-19.1.1-326.g8921c8ef.e 5.9 MB/s | 506 kB 00:00 2024-09-16T05:18:35.075 INFO:teuthology.orchestra.run.smithi178.stdout:(5/32): ceph-radosgw-19.1.1-326.g8921c8ef.el9.x 11 MB/s | 11 MB 00:00 2024-09-16T05:18:35.133 INFO:teuthology.orchestra.run.smithi178.stdout:(6/32): python3-ceph-argparse-19.1.1-326.g8921c 776 kB/s | 45 kB 00:00 2024-09-16T05:18:35.350 INFO:teuthology.orchestra.run.smithi178.stdout:(7/32): ceph-common-19.1.1-326.g8921c8ef.el9.x8 18 MB/s | 21 MB 00:01 2024-09-16T05:18:35.376 INFO:teuthology.orchestra.run.smithi178.stdout:(8/32): python3-ceph-common-19.1.1-326.g8921c8e 563 kB/s | 136 kB 00:00 2024-09-16T05:18:35.409 INFO:teuthology.orchestra.run.smithi178.stdout:(9/32): python3-cephfs-19.1.1-326.g8921c8ef.el9 2.7 MB/s | 161 kB 00:00 2024-09-16T05:18:35.435 INFO:teuthology.orchestra.run.smithi178.stdout:(10/32): python3-rados-19.1.1-326.g8921c8ef.el9 5.3 MB/s | 320 kB 00:00 2024-09-16T05:18:35.460 INFO:teuthology.orchestra.run.smithi178.stdout:(11/32): python3-rbd-19.1.1-326.g8921c8ef.el9.x 5.8 MB/s | 300 kB 00:00 2024-09-16T05:18:35.485 INFO:teuthology.orchestra.run.smithi178.stdout:(12/32): python3-rgw-19.1.1-326.g8921c8ef.el9.x 1.9 MB/s | 99 kB 00:00 2024-09-16T05:18:35.569 INFO:teuthology.orchestra.run.smithi178.stdout:(13/32): librgw2-19.1.1-326.g8921c8ef.el9.x86_6 8.6 MB/s | 5.3 MB 00:00 2024-09-16T05:18:35.619 INFO:teuthology.orchestra.run.smithi178.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 673 kB/s | 107 kB 00:00 2024-09-16T05:18:35.653 INFO:teuthology.orchestra.run.smithi178.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 541 kB/s | 45 kB 00:00 2024-09-16T05:18:35.678 INFO:teuthology.orchestra.run.smithi178.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 832 kB/s | 160 kB 00:00 2024-09-16T05:18:35.703 INFO:teuthology.orchestra.run.smithi178.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 662 kB 00:00 2024-09-16T05:18:35.728 INFO:teuthology.orchestra.run.smithi178.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 843 kB/s | 42 kB 00:00 2024-09-16T05:18:35.754 INFO:teuthology.orchestra.run.smithi178.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-09-16T05:18:35.787 INFO:teuthology.orchestra.run.smithi178.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 221 kB 00:00 2024-09-16T05:18:35.813 INFO:teuthology.orchestra.run.smithi178.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 428 kB/s | 36 kB 00:00 2024-09-16T05:18:35.838 INFO:teuthology.orchestra.run.smithi178.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 945 kB/s | 47 kB 00:00 2024-09-16T05:18:35.897 INFO:teuthology.orchestra.run.smithi178.stdout:(23/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.8 MB/s | 308 kB 00:00 2024-09-16T05:18:35.930 INFO:teuthology.orchestra.run.smithi178.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.5 MB/s | 49 kB 00:00 2024-09-16T05:18:35.955 INFO:teuthology.orchestra.run.smithi178.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 251 kB/s | 26 kB 00:00 2024-09-16T05:18:35.981 INFO:teuthology.orchestra.run.smithi178.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-16T05:18:36.014 INFO:teuthology.orchestra.run.smithi178.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-16T05:18:36.073 INFO:teuthology.orchestra.run.smithi178.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.0 MB/s | 839 kB 00:00 2024-09-16T05:18:36.140 INFO:teuthology.orchestra.run.smithi178.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-09-16T05:18:36.207 INFO:teuthology.orchestra.run.smithi178.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-16T05:18:36.315 INFO:teuthology.orchestra.run.smithi178.stdout:(31/32): librados2-19.1.1-326.g8921c8ef.el9.x86 14 MB/s | 3.4 MB 00:00 2024-09-16T05:18:36.407 INFO:teuthology.orchestra.run.smithi178.stdout:(32/32): librbd1-19.1.1-326.g8921c8ef.el9.x86_6 12 MB/s | 3.2 MB 00:00 2024-09-16T05:18:36.412 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:18:36.412 INFO:teuthology.orchestra.run.smithi178.stdout:Total 23 MB/s | 60 MB 00:02 2024-09-16T05:18:36.514 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-09-16T05:18:36.549 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-09-16T05:18:36.550 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-09-16T05:18:37.214 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-09-16T05:18:37.214 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-09-16T05:18:38.716 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-09-16T05:18:38.791 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-16T05:18:38.820 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-16T05:18:38.854 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-16T05:18:38.877 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-16T05:18:38.897 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-16T05:18:39.087 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-16T05:18:39.140 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-16T05:18:39.542 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-16T05:18:39.606 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T05:18:39.698 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T05:18:39.787 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 8/34 2024-09-16T05:18:39.815 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/34 2024-09-16T05:18:39.870 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/34 2024-09-16T05:18:39.912 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-16T05:18:39.977 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x 11/34 2024-09-16T05:18:40.076 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 12/34 2024-09-16T05:18:40.110 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T05:18:40.174 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T05:18:40.208 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-16T05:18:40.324 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-16T05:18:40.359 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-16T05:18:40.399 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-16T05:18:40.440 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-16T05:18:40.496 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-16T05:18:40.545 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-16T05:18:40.859 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-16T05:18:40.907 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 22/34 2024-09-16T05:18:40.996 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 22/34 2024-09-16T05:18:41.029 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 23/34 2024-09-16T05:18:41.064 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-16T05:18:41.543 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-16T05:18:41.568 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 26/34 2024-09-16T05:18:41.639 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 26/34 2024-09-16T05:18:41.707 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 27/34 2024-09-16T05:18:41.746 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86 28/34 2024-09-16T05:18:44.045 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T05:18:44.083 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T05:18:44.671 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T05:18:44.711 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 30/34 2024-09-16T05:18:44.804 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 30/34 2024-09-16T05:18:44.804 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T05:18:44.804 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T05:18:44.804 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:18:44.838 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T05:18:55.136 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T05:18:55.137 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /sys 2024-09-16T05:18:55.137 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /proc 2024-09-16T05:18:55.137 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /mnt 2024-09-16T05:18:55.137 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /var/tmp 2024-09-16T05:18:55.137 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /home 2024-09-16T05:18:55.137 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /root 2024-09-16T05:18:55.137 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /tmp 2024-09-16T05:18:55.137 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:18:56.138 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 32/34 2024-09-16T05:18:56.163 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 32/34 2024-09-16T05:18:56.163 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T05:18:56.163 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T05:18:56.163 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T05:18:56.163 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T05:18:56.163 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:18:56.165 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-16T05:18:56.248 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-16T05:18:56.248 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T05:18:57.456 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T05:18:57.456 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 1/34 2024-09-16T05:18:57.456 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2/34 2024-09-16T05:18:57.456 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 3/34 2024-09-16T05:18:57.456 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 4/34 2024-09-16T05:18:57.456 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 5/34 2024-09-16T05:18:57.457 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 6/34 2024-09-16T05:18:57.457 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T05:18:57.457 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x 8/34 2024-09-16T05:18:57.457 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86 9/34 2024-09-16T05:18:57.457 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 10/34 2024-09-16T05:18:57.457 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 11/34 2024-09-16T05:18:57.457 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 12/34 2024-09-16T05:18:57.457 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T05:18:57.457 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-16T05:18:57.457 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-16T05:18:57.457 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-16T05:18:57.458 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-16T05:18:57.458 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-16T05:18:57.458 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-16T05:18:57.458 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-16T05:18:57.458 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-16T05:18:57.458 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-16T05:18:57.458 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-16T05:18:57.458 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-16T05:18:57.458 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-16T05:18:57.458 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-16T05:18:57.458 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-16T05:18:57.458 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-16T05:18:57.459 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-16T05:18:57.459 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-16T05:18:57.459 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T05:18:57.459 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-16T05:18:57.459 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 33/34 2024-09-16T05:18:58.357 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T05:18:58.357 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:18:58.357 INFO:teuthology.orchestra.run.smithi178.stdout:Upgraded: 2024-09-16T05:18:58.357 INFO:teuthology.orchestra.run.smithi178.stdout: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T05:18:58.357 INFO:teuthology.orchestra.run.smithi178.stdout: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T05:18:58.357 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-09-16T05:18:58.357 INFO:teuthology.orchestra.run.smithi178.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T05:18:58.357 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T05:18:58.357 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T05:18:58.357 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T05:18:58.357 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T05:18:58.358 INFO:teuthology.orchestra.run.smithi178.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T05:18:58.358 INFO:teuthology.orchestra.run.smithi178.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T05:18:58.358 INFO:teuthology.orchestra.run.smithi178.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T05:18:58.358 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T05:18:58.358 INFO:teuthology.orchestra.run.smithi178.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T05:18:58.358 INFO:teuthology.orchestra.run.smithi178.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T05:18:58.358 INFO:teuthology.orchestra.run.smithi178.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T05:18:58.358 INFO:teuthology.orchestra.run.smithi178.stdout: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T05:18:58.358 INFO:teuthology.orchestra.run.smithi178.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T05:18:58.358 INFO:teuthology.orchestra.run.smithi178.stdout: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T05:18:58.358 INFO:teuthology.orchestra.run.smithi178.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T05:18:58.358 INFO:teuthology.orchestra.run.smithi178.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T05:18:58.358 INFO:teuthology.orchestra.run.smithi178.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T05:18:58.359 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T05:18:58.359 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T05:18:58.359 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T05:18:58.359 INFO:teuthology.orchestra.run.smithi178.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T05:18:58.359 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T05:18:58.359 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T05:18:58.359 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T05:18:58.359 INFO:teuthology.orchestra.run.smithi178.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T05:18:58.359 INFO:teuthology.orchestra.run.smithi178.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T05:18:58.359 INFO:teuthology.orchestra.run.smithi178.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T05:18:58.359 INFO:teuthology.orchestra.run.smithi178.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T05:18:58.359 INFO:teuthology.orchestra.run.smithi178.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T05:18:58.359 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:18:58.360 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-09-16T05:18:58.712 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-test 2024-09-16T05:18:59.279 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:00:28 ago on Mon 16 Sep 2024 05:18:31 AM UTC. 2024-09-16T05:18:59.419 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-09-16T05:18:59.420 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:18:59.420 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-09-16T05:18:59.420 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:18:59.420 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-09-16T05:18:59.420 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-test x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 49 M 2024-09-16T05:18:59.420 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-09-16T05:18:59.421 INFO:teuthology.orchestra.run.smithi178.stdout: libcephsqlite x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 163 k 2024-09-16T05:18:59.421 INFO:teuthology.orchestra.run.smithi178.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T05:18:59.421 INFO:teuthology.orchestra.run.smithi178.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T05:18:59.421 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:18:59.421 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-09-16T05:18:59.421 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:18:59.421 INFO:teuthology.orchestra.run.smithi178.stdout:Install 4 Packages 2024-09-16T05:18:59.421 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:18:59.422 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 49 M 2024-09-16T05:18:59.422 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 209 M 2024-09-16T05:18:59.422 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-09-16T05:18:59.912 INFO:teuthology.orchestra.run.smithi178.stdout:(1/4): libcephsqlite-19.1.1-326.g8921c8ef.el9.x 771 kB/s | 163 kB 00:00 2024-09-16T05:19:00.287 INFO:teuthology.orchestra.run.smithi178.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 170 kB/s | 64 kB 00:00 2024-09-16T05:19:00.337 INFO:teuthology.orchestra.run.smithi178.stdout:(3/4): socat-1.7.4.1-6.el9.x86_64.rpm 477 kB/s | 304 kB 00:00 2024-09-16T05:19:01.237 INFO:teuthology.orchestra.run.smithi178.stdout:(4/4): ceph-test-19.1.1-326.g8921c8ef.el9.x86_6 32 MB/s | 49 MB 00:01 2024-09-16T05:19:01.239 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:19:01.240 INFO:teuthology.orchestra.run.smithi178.stdout:Total 27 MB/s | 49 MB 00:01 2024-09-16T05:19:01.267 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-09-16T05:19:01.287 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-09-16T05:19:01.288 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-09-16T05:19:01.673 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-09-16T05:19:01.673 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-09-16T05:19:02.204 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-09-16T05:19:02.287 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T05:19:02.336 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T05:19:02.364 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 3/4 2024-09-16T05:19:06.956 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 3/4 2024-09-16T05:19:07.037 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 4/4 2024-09-16T05:19:09.247 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 4/4 2024-09-16T05:19:09.247 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 1/4 2024-09-16T05:19:09.247 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2/4 2024-09-16T05:19:09.247 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T05:19:09.544 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T05:19:09.544 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:19:09.544 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-09-16T05:19:09.544 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T05:19:09.544 INFO:teuthology.orchestra.run.smithi178.stdout: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T05:19:09.544 INFO:teuthology.orchestra.run.smithi178.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T05:19:09.545 INFO:teuthology.orchestra.run.smithi178.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T05:19:09.545 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:19:09.545 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-09-16T05:19:09.825 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph 2024-09-16T05:19:10.399 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:00:39 ago on Mon 16 Sep 2024 05:18:31 AM UTC. 2024-09-16T05:19:10.544 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-09-16T05:19:10.547 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:19:10.547 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-09-16T05:19:10.547 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:19:10.547 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-09-16T05:19:10.547 INFO:teuthology.orchestra.run.smithi178.stdout: ceph x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 6.5 k 2024-09-16T05:19:10.547 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-09-16T05:19:10.548 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mds x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 2.4 M 2024-09-16T05:19:10.548 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 1.5 M 2024-09-16T05:19:10.548 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-modules-core noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 249 k 2024-09-16T05:19:10.548 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mon x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 4.7 M 2024-09-16T05:19:10.548 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-osd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 17 M 2024-09-16T05:19:10.548 INFO:teuthology.orchestra.run.smithi178.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T05:19:10.548 INFO:teuthology.orchestra.run.smithi178.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T05:19:10.548 INFO:teuthology.orchestra.run.smithi178.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T05:19:10.548 INFO:teuthology.orchestra.run.smithi178.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T05:19:10.548 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T05:19:10.548 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T05:19:10.548 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T05:19:10.548 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T05:19:10.549 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T05:19:10.549 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T05:19:10.549 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-16T05:19:10.549 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T05:19:10.549 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T05:19:10.549 INFO:teuthology.orchestra.run.smithi178.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T05:19:10.549 INFO:teuthology.orchestra.run.smithi178.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T05:19:10.549 INFO:teuthology.orchestra.run.smithi178.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T05:19:10.549 INFO:teuthology.orchestra.run.smithi178.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T05:19:10.549 INFO:teuthology.orchestra.run.smithi178.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T05:19:10.549 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T05:19:10.549 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T05:19:10.549 INFO:teuthology.orchestra.run.smithi178.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T05:19:10.550 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T05:19:10.550 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T05:19:10.550 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T05:19:10.550 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T05:19:10.550 INFO:teuthology.orchestra.run.smithi178.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T05:19:10.550 INFO:teuthology.orchestra.run.smithi178.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T05:19:10.550 INFO:teuthology.orchestra.run.smithi178.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T05:19:10.550 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T05:19:10.550 INFO:teuthology.orchestra.run.smithi178.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T05:19:10.550 INFO:teuthology.orchestra.run.smithi178.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T05:19:10.550 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:19:10.550 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-09-16T05:19:10.550 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:19:10.550 INFO:teuthology.orchestra.run.smithi178.stdout:Install 37 Packages 2024-09-16T05:19:10.551 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:19:10.551 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 30 M 2024-09-16T05:19:10.551 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 108 M 2024-09-16T05:19:10.551 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-09-16T05:19:11.454 INFO:teuthology.orchestra.run.smithi178.stdout:(1/37): ceph-19.1.1-326.g8921c8ef.el9.x86_64.rp 50 kB/s | 6.5 kB 00:00 2024-09-16T05:19:11.638 INFO:teuthology.orchestra.run.smithi178.stdout:(2/37): ceph-mgr-19.1.1-326.g8921c8ef.el9.x86_6 4.8 MB/s | 1.5 MB 00:00 2024-09-16T05:19:11.696 INFO:teuthology.orchestra.run.smithi178.stdout:(3/37): ceph-mds-19.1.1-326.g8921c8ef.el9.x86_6 6.4 MB/s | 2.4 MB 00:00 2024-09-16T05:19:11.780 INFO:teuthology.orchestra.run.smithi178.stdout:(4/37): ceph-mgr-modules-core-19.1.1-326.g8921c 2.9 MB/s | 249 kB 00:00 2024-09-16T05:19:11.888 INFO:teuthology.orchestra.run.smithi178.stdout:(5/37): ceph-mon-19.1.1-326.g8921c8ef.el9.x86_6 11 MB/s | 4.7 MB 00:00 2024-09-16T05:19:11.956 INFO:teuthology.orchestra.run.smithi178.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.4 MB/s | 253 kB 00:00 2024-09-16T05:19:12.031 INFO:teuthology.orchestra.run.smithi178.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 106 kB 00:00 2024-09-16T05:19:12.206 INFO:teuthology.orchestra.run.smithi178.stdout:(8/37): ceph-osd-19.1.1-326.g8921c8ef.el9.x86_6 30 MB/s | 17 MB 00:00 2024-09-16T05:19:12.231 INFO:teuthology.orchestra.run.smithi178.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 675 kB/s | 135 kB 00:00 2024-09-16T05:19:12.265 INFO:teuthology.orchestra.run.smithi178.stdout:(10/37): python3-cryptography-36.0.1-4.el9.x86_ 3.3 MB/s | 1.2 MB 00:00 2024-09-16T05:19:12.307 INFO:teuthology.orchestra.run.smithi178.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 2.8 MB/s | 215 kB 00:00 2024-09-16T05:19:12.344 INFO:teuthology.orchestra.run.smithi178.stdout:(12/37): python3-requests-2.25.1-8.el9.noarch.r 910 kB/s | 125 kB 00:00 2024-09-16T05:19:12.386 INFO:teuthology.orchestra.run.smithi178.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 188 kB 00:00 2024-09-16T05:19:12.436 INFO:teuthology.orchestra.run.smithi178.stdout:(14/37): python3-packaging-20.9-5.el9.noarch.rp 1.5 MB/s | 77 kB 00:00 2024-09-16T05:19:12.453 INFO:teuthology.orchestra.run.smithi178.stdout:(15/37): python3-markupsafe-1.1.1-12.el9.x86_64 320 kB/s | 35 kB 00:00 2024-09-16T05:19:12.475 INFO:teuthology.orchestra.run.smithi178.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.3 MB/s | 51 kB 00:00 2024-09-16T05:19:12.496 INFO:teuthology.orchestra.run.smithi178.stdout:(17/37): python3-mako-1.1.4-6.el9.noarch.rpm 915 kB/s | 172 kB 00:00 2024-09-16T05:19:12.512 INFO:teuthology.orchestra.run.smithi178.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 705 kB/s | 42 kB 00:00 2024-09-16T05:19:12.554 INFO:teuthology.orchestra.run.smithi178.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-09-16T05:19:12.580 INFO:teuthology.orchestra.run.smithi178.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 214 kB/s | 22 kB 00:00 2024-09-16T05:19:12.605 INFO:teuthology.orchestra.run.smithi178.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 471 kB/s | 43 kB 00:00 2024-09-16T05:19:12.639 INFO:teuthology.orchestra.run.smithi178.stdout:(22/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-09-16T05:19:12.664 INFO:teuthology.orchestra.run.smithi178.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 206 kB/s | 11 kB 00:00 2024-09-16T05:19:12.689 INFO:teuthology.orchestra.run.smithi178.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 364 kB/s | 18 kB 00:00 2024-09-16T05:19:12.714 INFO:teuthology.orchestra.run.smithi178.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2024-09-16T05:19:12.739 INFO:teuthology.orchestra.run.smithi178.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-16T05:19:12.765 INFO:teuthology.orchestra.run.smithi178.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-16T05:19:12.790 INFO:teuthology.orchestra.run.smithi178.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 922 kB/s | 46 kB 00:00 2024-09-16T05:19:12.824 INFO:teuthology.orchestra.run.smithi178.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-09-16T05:19:12.866 INFO:teuthology.orchestra.run.smithi178.stdout:(30/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.2 MB/s | 358 kB 00:00 2024-09-16T05:19:12.891 INFO:teuthology.orchestra.run.smithi178.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 244 kB/s | 16 kB 00:00 2024-09-16T05:19:12.916 INFO:teuthology.orchestra.run.smithi178.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-16T05:19:12.941 INFO:teuthology.orchestra.run.smithi178.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 715 kB/s | 36 kB 00:00 2024-09-16T05:19:12.967 INFO:teuthology.orchestra.run.smithi178.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.5 MB/s | 272 kB 00:00 2024-09-16T05:19:12.992 INFO:teuthology.orchestra.run.smithi178.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 796 kB/s | 20 kB 00:00 2024-09-16T05:19:13.017 INFO:teuthology.orchestra.run.smithi178.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-16T05:19:13.043 INFO:teuthology.orchestra.run.smithi178.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-09-16T05:19:13.046 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:19:13.046 INFO:teuthology.orchestra.run.smithi178.stdout:Total 12 MB/s | 30 MB 00:02 2024-09-16T05:19:13.188 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-09-16T05:19:13.228 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-09-16T05:19:13.228 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-09-16T05:19:13.709 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-09-16T05:19:13.709 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-09-16T05:19:15.057 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-09-16T05:19:15.120 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-16T05:19:15.171 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-16T05:19:15.206 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-16T05:19:15.276 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-16T05:19:15.321 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-16T05:19:15.355 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-16T05:19:15.446 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-16T05:19:15.515 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-16T05:19:15.557 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-16T05:19:15.600 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-16T05:19:15.630 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-16T05:19:15.665 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-16T05:19:15.698 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-16T05:19:15.747 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-16T05:19:15.790 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-16T05:19:15.825 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-16T05:19:15.873 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-16T05:19:15.914 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-16T05:19:15.995 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-16T05:19:16.140 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-16T05:19:16.229 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-16T05:19:16.283 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-16T05:19:16.322 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-16T05:19:16.371 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-16T05:19:16.434 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-16T05:19:16.626 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-16T05:19:16.668 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-16T05:19:16.746 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-16T05:19:16.835 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-16T05:19:16.886 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-16T05:19:17.007 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-16T05:19:17.173 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.n 32/37 2024-09-16T05:19:17.202 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 33/37 2024-09-16T05:19:17.228 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 33/37 2024-09-16T05:19:17.228 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T05:19:17.229 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T05:19:17.229 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T05:19:17.229 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T05:19:17.229 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:19:18.670 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 34/37 2024-09-16T05:19:18.697 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 34/37 2024-09-16T05:19:18.697 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T05:19:18.697 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T05:19:18.697 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T05:19:18.697 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T05:19:18.697 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:19:19.129 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 35/37 2024-09-16T05:19:19.153 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 35/37 2024-09-16T05:19:19.153 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T05:19:19.153 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T05:19:19.153 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T05:19:19.153 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T05:19:19.153 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:19:19.399 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 36/37 2024-09-16T05:19:19.423 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 36/37 2024-09-16T05:19:19.423 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T05:19:19.423 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T05:19:19.423 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T05:19:19.423 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T05:19:19.424 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:19:19.493 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 37/37 2024-09-16T05:19:21.247 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 37/37 2024-09-16T05:19:21.247 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 1/37 2024-09-16T05:19:21.247 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2/37 2024-09-16T05:19:21.247 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 3/37 2024-09-16T05:19:21.248 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 4/37 2024-09-16T05:19:21.248 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 5/37 2024-09-16T05:19:21.248 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.n 6/37 2024-09-16T05:19:21.248 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-16T05:19:21.248 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-16T05:19:21.248 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-16T05:19:21.248 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-16T05:19:21.248 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-16T05:19:21.248 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-16T05:19:21.248 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-16T05:19:21.248 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-16T05:19:21.248 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-16T05:19:21.248 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-16T05:19:21.249 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-16T05:19:21.249 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-16T05:19:21.249 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-16T05:19:21.249 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-16T05:19:21.249 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-16T05:19:21.249 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-16T05:19:21.249 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-16T05:19:21.249 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-16T05:19:21.249 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-16T05:19:21.249 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-16T05:19:21.249 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-16T05:19:21.250 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-16T05:19:21.250 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-16T05:19:21.250 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-16T05:19:21.251 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-16T05:19:21.251 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-16T05:19:21.251 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-16T05:19:21.251 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-16T05:19:21.251 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-16T05:19:21.251 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-16T05:19:22.239 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-16T05:19:22.240 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:19:22.240 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-09-16T05:19:22.240 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T05:19:22.240 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T05:19:22.240 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T05:19:22.240 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T05:19:22.240 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T05:19:22.240 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T05:19:22.240 INFO:teuthology.orchestra.run.smithi178.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T05:19:22.240 INFO:teuthology.orchestra.run.smithi178.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T05:19:22.240 INFO:teuthology.orchestra.run.smithi178.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T05:19:22.241 INFO:teuthology.orchestra.run.smithi178.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T05:19:22.241 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T05:19:22.241 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T05:19:22.241 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T05:19:22.241 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T05:19:22.241 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T05:19:22.241 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T05:19:22.241 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T05:19:22.241 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T05:19:22.241 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T05:19:22.241 INFO:teuthology.orchestra.run.smithi178.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T05:19:22.241 INFO:teuthology.orchestra.run.smithi178.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T05:19:22.242 INFO:teuthology.orchestra.run.smithi178.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T05:19:22.242 INFO:teuthology.orchestra.run.smithi178.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T05:19:22.242 INFO:teuthology.orchestra.run.smithi178.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T05:19:22.242 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T05:19:22.242 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T05:19:22.242 INFO:teuthology.orchestra.run.smithi178.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T05:19:22.242 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T05:19:22.242 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T05:19:22.242 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T05:19:22.242 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T05:19:22.242 INFO:teuthology.orchestra.run.smithi178.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T05:19:22.242 INFO:teuthology.orchestra.run.smithi178.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T05:19:22.242 INFO:teuthology.orchestra.run.smithi178.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T05:19:22.243 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T05:19:22.243 INFO:teuthology.orchestra.run.smithi178.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T05:19:22.243 INFO:teuthology.orchestra.run.smithi178.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T05:19:22.243 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:19:22.243 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-09-16T05:19:22.590 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-base 2024-09-16T05:19:23.167 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:00:52 ago on Mon 16 Sep 2024 05:18:31 AM UTC. 2024-09-16T05:19:23.257 INFO:teuthology.orchestra.run.smithi178.stdout:Package ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T05:19:23.299 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-09-16T05:19:23.300 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-09-16T05:19:23.300 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-09-16T05:19:23.365 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install cephadm 2024-09-16T05:19:23.924 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:00:52 ago on Mon 16 Sep 2024 05:18:31 AM UTC. 2024-09-16T05:19:24.055 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-09-16T05:19:24.056 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:19:24.056 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-09-16T05:19:24.056 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:19:24.056 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-09-16T05:19:24.056 INFO:teuthology.orchestra.run.smithi178.stdout: cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 765 k 2024-09-16T05:19:24.056 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:19:24.057 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-09-16T05:19:24.057 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:19:24.057 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-09-16T05:19:24.057 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:19:24.057 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 765 k 2024-09-16T05:19:24.057 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 771 k 2024-09-16T05:19:24.057 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-09-16T05:19:24.314 INFO:teuthology.orchestra.run.smithi178.stdout:cephadm-19.1.1-326.g8921c8ef.el9.noarch.rpm 2.9 MB/s | 765 kB 00:00 2024-09-16T05:19:24.315 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:19:24.315 INFO:teuthology.orchestra.run.smithi178.stdout:Total 2.9 MB/s | 765 kB 00:00 2024-09-16T05:19:24.315 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-09-16T05:19:24.321 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-09-16T05:19:24.321 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-09-16T05:19:24.327 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-09-16T05:19:24.327 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-09-16T05:19:24.400 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-09-16T05:19:24.824 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T05:19:24.901 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T05:19:25.284 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T05:19:25.527 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T05:19:25.527 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:19:25.527 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-09-16T05:19:25.527 INFO:teuthology.orchestra.run.smithi178.stdout: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T05:19:25.527 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:19:25.528 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-09-16T05:19:25.734 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T05:19:26.314 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:00:55 ago on Mon 16 Sep 2024 05:18:31 AM UTC. 2024-09-16T05:19:26.450 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-09-16T05:19:26.450 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:19:26.450 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-09-16T05:19:26.450 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:19:26.450 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-09-16T05:19:26.450 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 145 k 2024-09-16T05:19:26.450 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:19:26.450 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-09-16T05:19:26.450 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:19:26.451 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-09-16T05:19:26.451 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:19:26.451 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 145 k 2024-09-16T05:19:26.451 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 435 k 2024-09-16T05:19:26.451 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-09-16T05:19:26.662 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-immutable-object-cache-19.1.1-326.g8921c8e 691 kB/s | 145 kB 00:00 2024-09-16T05:19:26.662 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:19:26.663 INFO:teuthology.orchestra.run.smithi178.stdout:Total 685 kB/s | 145 kB 00:00 2024-09-16T05:19:26.663 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-09-16T05:19:26.670 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-09-16T05:19:26.670 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-09-16T05:19:26.726 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-09-16T05:19:26.727 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-09-16T05:19:26.891 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-09-16T05:19:26.988 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T05:19:27.013 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T05:19:27.013 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T05:19:27.013 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T05:19:27.013 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:19:27.946 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T05:19:27.946 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:19:27.946 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-09-16T05:19:27.946 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T05:19:27.946 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:19:27.946 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-09-16T05:19:28.176 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr 2024-09-16T05:19:28.754 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:00:57 ago on Mon 16 Sep 2024 05:18:31 AM UTC. 2024-09-16T05:19:28.847 INFO:teuthology.orchestra.run.smithi178.stdout:Package ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T05:19:28.889 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-09-16T05:19:28.890 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-09-16T05:19:28.890 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-09-16T05:19:28.957 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T05:19:29.519 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:00:58 ago on Mon 16 Sep 2024 05:18:31 AM UTC. 2024-09-16T05:19:29.652 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-09-16T05:19:29.653 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:19:29.653 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-09-16T05:19:29.653 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:19:29.653 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-09-16T05:19:29.653 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-dashboard noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 3.6 M 2024-09-16T05:19:29.653 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-09-16T05:19:29.653 INFO:teuthology.orchestra.run.smithi178.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T05:19:29.653 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-grafana-dashboards noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 32 k 2024-09-16T05:19:29.653 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-prometheus-alerts noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 16 k 2024-09-16T05:19:29.653 INFO:teuthology.orchestra.run.smithi178.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T05:19:29.654 INFO:teuthology.orchestra.run.smithi178.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T05:19:29.654 INFO:teuthology.orchestra.run.smithi178.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T05:19:29.654 INFO:teuthology.orchestra.run.smithi178.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T05:19:29.654 INFO:teuthology.orchestra.run.smithi178.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T05:19:29.654 INFO:teuthology.orchestra.run.smithi178.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T05:19:29.654 INFO:teuthology.orchestra.run.smithi178.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T05:19:29.654 INFO:teuthology.orchestra.run.smithi178.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T05:19:29.654 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:19:29.654 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-09-16T05:19:29.654 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:19:29.654 INFO:teuthology.orchestra.run.smithi178.stdout:Install 12 Packages 2024-09-16T05:19:29.654 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:19:29.655 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 8.7 M 2024-09-16T05:19:29.655 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 104 M 2024-09-16T05:19:29.655 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-09-16T05:19:30.100 INFO:teuthology.orchestra.run.smithi178.stdout:(1/12): ceph-grafana-dashboards-19.1.1-326.g892 224 kB/s | 32 kB 00:00 2024-09-16T05:19:30.115 INFO:teuthology.orchestra.run.smithi178.stdout:(2/12): ceph-prometheus-alerts-19.1.1-326.g8921 103 kB/s | 16 kB 00:00 2024-09-16T05:19:30.307 INFO:teuthology.orchestra.run.smithi178.stdout:(3/12): ceph-mgr-dashboard-19.1.1-326.g8921c8ef 10 MB/s | 3.6 MB 00:00 2024-09-16T05:19:30.332 INFO:teuthology.orchestra.run.smithi178.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.2 MB/s | 269 kB 00:00 2024-09-16T05:19:30.408 INFO:teuthology.orchestra.run.smithi178.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 7.1 MB/s | 548 kB 00:00 2024-09-16T05:19:30.450 INFO:teuthology.orchestra.run.smithi178.stdout:(6/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.9 MB/s | 1.0 MB 00:00 2024-09-16T05:19:30.466 INFO:teuthology.orchestra.run.smithi178.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 333 kB/s | 19 kB 00:00 2024-09-16T05:19:30.500 INFO:teuthology.orchestra.run.smithi178.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.3 MB/s | 144 kB 00:00 2024-09-16T05:19:30.517 INFO:teuthology.orchestra.run.smithi178.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 2.0 MB/s | 31 kB 00:00 2024-09-16T05:19:30.551 INFO:teuthology.orchestra.run.smithi178.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 5.5 MB/s | 188 kB 00:00 2024-09-16T05:19:30.667 INFO:teuthology.orchestra.run.smithi178.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 9.4 MB/s | 2.0 MB 00:00 2024-09-16T05:19:30.968 INFO:teuthology.orchestra.run.smithi178.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 1.3 MB/s | 863 kB 00:00 2024-09-16T05:19:30.971 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:19:30.972 INFO:teuthology.orchestra.run.smithi178.stdout:Total 6.6 MB/s | 8.7 MB 00:01 2024-09-16T05:19:31.020 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-09-16T05:19:31.037 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-09-16T05:19:31.037 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-09-16T05:19:31.236 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-09-16T05:19:31.237 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-09-16T05:19:31.758 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-09-16T05:19:31.835 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-16T05:19:31.870 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-16T05:19:31.917 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-16T05:19:31.956 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-16T05:19:32.047 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-16T05:19:32.173 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-16T05:19:32.259 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-16T05:19:32.311 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-16T05:19:32.347 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-16T05:19:32.379 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9. 10/12 2024-09-16T05:19:33.434 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9 11/12 2024-09-16T05:19:33.472 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 12/12 2024-09-16T05:19:34.892 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 12/12 2024-09-16T05:19:34.892 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9 1/12 2024-09-16T05:19:34.892 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 2/12 2024-09-16T05:19:34.893 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9. 3/12 2024-09-16T05:19:34.893 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-16T05:19:34.893 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-16T05:19:34.893 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-16T05:19:34.893 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-16T05:19:34.893 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-16T05:19:34.893 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-16T05:19:34.893 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-16T05:19:34.893 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-16T05:19:35.204 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-16T05:19:35.204 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:19:35.204 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-09-16T05:19:35.204 INFO:teuthology.orchestra.run.smithi178.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T05:19:35.204 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T05:19:35.204 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T05:19:35.204 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T05:19:35.205 INFO:teuthology.orchestra.run.smithi178.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T05:19:35.205 INFO:teuthology.orchestra.run.smithi178.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T05:19:35.205 INFO:teuthology.orchestra.run.smithi178.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T05:19:35.205 INFO:teuthology.orchestra.run.smithi178.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T05:19:35.205 INFO:teuthology.orchestra.run.smithi178.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T05:19:35.205 INFO:teuthology.orchestra.run.smithi178.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T05:19:35.205 INFO:teuthology.orchestra.run.smithi178.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T05:19:35.205 INFO:teuthology.orchestra.run.smithi178.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T05:19:35.205 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:19:35.205 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-09-16T05:19:35.454 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T05:19:36.015 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:05 ago on Mon 16 Sep 2024 05:18:31 AM UTC. 2024-09-16T05:19:36.148 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-09-16T05:19:36.148 INFO:teuthology.orchestra.run.smithi178.stdout:====================================================================================== 2024-09-16T05:19:36.149 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-09-16T05:19:36.149 INFO:teuthology.orchestra.run.smithi178.stdout:====================================================================================== 2024-09-16T05:19:36.149 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-09-16T05:19:36.149 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 7.4 M 2024-09-16T05:19:36.149 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-09-16T05:19:36.149 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T05:19:36.149 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T05:19:36.149 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T05:19:36.149 INFO:teuthology.orchestra.run.smithi178.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T05:19:36.149 INFO:teuthology.orchestra.run.smithi178.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T05:19:36.149 INFO:teuthology.orchestra.run.smithi178.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T05:19:36.150 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T05:19:36.150 INFO:teuthology.orchestra.run.smithi178.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T05:19:36.150 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T05:19:36.150 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T05:19:36.150 INFO:teuthology.orchestra.run.smithi178.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T05:19:36.150 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:19:36.150 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-09-16T05:19:36.150 INFO:teuthology.orchestra.run.smithi178.stdout:====================================================================================== 2024-09-16T05:19:36.150 INFO:teuthology.orchestra.run.smithi178.stdout:Install 12 Packages 2024-09-16T05:19:36.150 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:19:36.151 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 38 M 2024-09-16T05:19:36.151 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 212 M 2024-09-16T05:19:36.151 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-09-16T05:19:36.992 INFO:teuthology.orchestra.run.smithi178.stdout:(1/12): ceph-mgr-diskprediction-local-19.1.1-32 16 MB/s | 7.4 MB 00:00 2024-09-16T05:19:37.101 INFO:teuthology.orchestra.run.smithi178.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 303 kB/s | 33 kB 00:00 2024-09-16T05:19:37.351 INFO:teuthology.orchestra.run.smithi178.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-09-16T05:19:37.376 INFO:teuthology.orchestra.run.smithi178.stdout:(4/12): libquadmath-11.5.0-2.el9.x86_64.rpm 226 kB/s | 191 kB 00:00 2024-09-16T05:19:37.393 INFO:teuthology.orchestra.run.smithi178.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 436 kB/s | 18 kB 00:00 2024-09-16T05:19:37.552 INFO:teuthology.orchestra.run.smithi178.stdout:(6/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 31 MB/s | 4.9 MB 00:00 2024-09-16T05:19:37.577 INFO:teuthology.orchestra.run.smithi178.stdout:(7/12): openblas-0.3.26-2.el9.x86_64.rpm 192 kB/s | 38 kB 00:00 2024-09-16T05:19:37.611 INFO:teuthology.orchestra.run.smithi178.stdout:(8/12): libgfortran-11.5.0-2.el9.x86_64.rpm 742 kB/s | 801 kB 00:01 2024-09-16T05:19:37.636 INFO:teuthology.orchestra.run.smithi178.stdout:(9/12): python3-devel-3.9.19-8.el9.x86_64.rpm 2.9 MB/s | 245 kB 00:00 2024-09-16T05:19:37.870 INFO:teuthology.orchestra.run.smithi178.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 969 kB/s | 250 kB 00:00 2024-09-16T05:19:37.945 INFO:teuthology.orchestra.run.smithi178.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 14 MB/s | 5.1 MB 00:00 2024-09-16T05:19:38.097 INFO:teuthology.orchestra.run.smithi178.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-09-16T05:19:38.097 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:19:38.097 INFO:teuthology.orchestra.run.smithi178.stdout:Total 20 MB/s | 38 MB 00:01 2024-09-16T05:19:38.451 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-09-16T05:19:38.477 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-09-16T05:19:38.477 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-09-16T05:19:38.900 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-09-16T05:19:38.900 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-09-16T05:19:39.665 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-09-16T05:19:39.737 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T05:19:39.779 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T05:19:39.883 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T05:19:39.919 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T05:19:40.104 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T05:19:40.136 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T05:19:40.218 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T05:19:40.879 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T05:19:41.005 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T05:19:42.237 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T05:19:43.262 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T05:19:43.292 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 12/12 2024-09-16T05:19:45.916 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 12/12 2024-09-16T05:19:45.916 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 1/12 2024-09-16T05:19:45.916 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T05:19:45.916 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T05:19:45.916 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T05:19:45.916 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T05:19:45.916 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T05:19:45.917 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T05:19:45.917 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T05:19:45.917 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T05:19:45.917 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T05:19:45.917 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T05:19:46.289 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T05:19:46.289 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:19:46.289 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-09-16T05:19:46.289 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T05:19:46.290 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T05:19:46.290 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T05:19:46.290 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T05:19:46.290 INFO:teuthology.orchestra.run.smithi178.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T05:19:46.290 INFO:teuthology.orchestra.run.smithi178.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T05:19:46.290 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T05:19:46.290 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T05:19:46.290 INFO:teuthology.orchestra.run.smithi178.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T05:19:46.290 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T05:19:46.290 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T05:19:46.290 INFO:teuthology.orchestra.run.smithi178.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T05:19:46.290 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:19:46.290 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-09-16T05:19:46.662 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-rook 2024-09-16T05:19:47.243 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:16 ago on Mon 16 Sep 2024 05:18:31 AM UTC. 2024-09-16T05:19:47.377 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-09-16T05:19:47.378 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:19:47.378 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-09-16T05:19:47.379 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:19:47.379 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-09-16T05:19:47.379 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-rook noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 49 k 2024-09-16T05:19:47.379 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-09-16T05:19:47.379 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T05:19:47.379 INFO:teuthology.orchestra.run.smithi178.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T05:19:47.379 INFO:teuthology.orchestra.run.smithi178.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T05:19:47.379 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T05:19:47.379 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T05:19:47.379 INFO:teuthology.orchestra.run.smithi178.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T05:19:47.379 INFO:teuthology.orchestra.run.smithi178.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T05:19:47.379 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T05:19:47.379 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T05:19:47.380 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T05:19:47.380 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T05:19:47.380 INFO:teuthology.orchestra.run.smithi178.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T05:19:47.380 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:19:47.380 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-09-16T05:19:47.380 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:19:47.380 INFO:teuthology.orchestra.run.smithi178.stdout:Install 13 Packages 2024-09-16T05:19:47.380 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:19:47.381 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 2.2 M 2024-09-16T05:19:47.381 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 26 M 2024-09-16T05:19:47.381 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-09-16T05:19:47.792 INFO:teuthology.orchestra.run.smithi178.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 160 kB/s | 19 kB 00:00 2024-09-16T05:19:47.817 INFO:teuthology.orchestra.run.smithi178.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 177 kB/s | 26 kB 00:00 2024-09-16T05:19:47.843 INFO:teuthology.orchestra.run.smithi178.stdout:(3/13): ceph-mgr-rook-19.1.1-326.g8921c8ef.el9. 289 kB/s | 49 kB 00:00 2024-09-16T05:19:47.901 INFO:teuthology.orchestra.run.smithi178.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-09-16T05:19:47.930 INFO:teuthology.orchestra.run.smithi178.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-09-16T05:19:47.960 INFO:teuthology.orchestra.run.smithi178.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 915 kB/s | 54 kB 00:00 2024-09-16T05:19:47.994 INFO:teuthology.orchestra.run.smithi178.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 480 kB/s | 32 kB 00:00 2024-09-16T05:19:48.019 INFO:teuthology.orchestra.run.smithi178.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 248 kB/s | 14 kB 00:00 2024-09-16T05:19:48.044 INFO:teuthology.orchestra.run.smithi178.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 4.4 MB/s | 220 kB 00:00 2024-09-16T05:19:48.069 INFO:teuthology.orchestra.run.smithi178.stdout:(10/13): python3-pyasn1-modules-0.4.8-6.el9.noa 1.2 MB/s | 279 kB 00:00 2024-09-16T05:19:48.095 INFO:teuthology.orchestra.run.smithi178.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-09-16T05:19:48.145 INFO:teuthology.orchestra.run.smithi178.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.2 MB/s | 90 kB 00:00 2024-09-16T05:19:48.178 INFO:teuthology.orchestra.run.smithi178.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.4 MB/s | 1.0 MB 00:00 2024-09-16T05:19:48.181 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:19:48.181 INFO:teuthology.orchestra.run.smithi178.stdout:Total 2.8 MB/s | 2.2 MB 00:00 2024-09-16T05:19:48.245 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-09-16T05:19:48.260 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-09-16T05:19:48.260 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-09-16T05:19:48.349 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-09-16T05:19:48.350 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-09-16T05:19:48.861 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-09-16T05:19:49.028 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T05:19:49.089 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T05:19:49.169 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T05:19:49.206 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T05:19:49.247 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T05:19:49.359 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T05:19:49.460 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T05:19:49.516 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T05:19:50.241 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T05:19:50.296 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T05:19:50.330 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T05:19:50.378 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T05:19:50.408 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 13/13 2024-09-16T05:19:51.222 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 13/13 2024-09-16T05:19:51.223 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 1/13 2024-09-16T05:19:51.223 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T05:19:51.223 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T05:19:51.223 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T05:19:51.223 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T05:19:51.223 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T05:19:51.223 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T05:19:51.223 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T05:19:51.223 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T05:19:51.223 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T05:19:51.223 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T05:19:51.223 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T05:19:51.551 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T05:19:51.551 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:19:51.551 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-09-16T05:19:51.551 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T05:19:51.551 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T05:19:51.551 INFO:teuthology.orchestra.run.smithi178.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T05:19:51.551 INFO:teuthology.orchestra.run.smithi178.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T05:19:51.551 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T05:19:51.551 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T05:19:51.552 INFO:teuthology.orchestra.run.smithi178.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T05:19:51.552 INFO:teuthology.orchestra.run.smithi178.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T05:19:51.552 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T05:19:51.552 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T05:19:51.552 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T05:19:51.552 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T05:19:51.552 INFO:teuthology.orchestra.run.smithi178.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T05:19:51.552 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:19:51.552 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-09-16T05:19:51.837 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T05:19:52.404 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:21 ago on Mon 16 Sep 2024 05:18:31 AM UTC. 2024-09-16T05:19:52.537 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-09-16T05:19:52.538 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:19:52.538 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-09-16T05:19:52.538 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:19:52.538 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-09-16T05:19:52.538 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 146 k 2024-09-16T05:19:52.538 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-09-16T05:19:52.538 INFO:teuthology.orchestra.run.smithi178.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T05:19:52.539 INFO:teuthology.orchestra.run.smithi178.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T05:19:52.539 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T05:19:52.539 INFO:teuthology.orchestra.run.smithi178.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T05:19:52.539 INFO:teuthology.orchestra.run.smithi178.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T05:19:52.539 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:19:52.539 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-09-16T05:19:52.539 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:19:52.539 INFO:teuthology.orchestra.run.smithi178.stdout:Install 6 Packages 2024-09-16T05:19:52.539 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:19:52.540 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 7.0 M 2024-09-16T05:19:52.540 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 33 M 2024-09-16T05:19:52.540 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-09-16T05:19:53.139 INFO:teuthology.orchestra.run.smithi178.stdout:(1/6): ceph-mgr-cephadm-19.1.1-326.g8921c8ef.el 690 kB/s | 146 kB 00:00 2024-09-16T05:19:53.198 INFO:teuthology.orchestra.run.smithi178.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 9.2 MB/s | 548 kB 00:00 2024-09-16T05:19:53.223 INFO:teuthology.orchestra.run.smithi178.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-16T05:19:53.249 INFO:teuthology.orchestra.run.smithi178.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-16T05:19:53.415 INFO:teuthology.orchestra.run.smithi178.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 509 kB/s | 248 kB 00:00 2024-09-16T05:19:55.657 INFO:teuthology.orchestra.run.smithi178.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 2.2 MB/s | 6.0 MB 00:02 2024-09-16T05:19:55.659 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:19:55.659 INFO:teuthology.orchestra.run.smithi178.stdout:Total 2.2 MB/s | 7.0 MB 00:03 2024-09-16T05:19:55.730 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-09-16T05:19:55.741 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-09-16T05:19:55.741 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-09-16T05:19:55.839 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-09-16T05:19:55.840 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-09-16T05:19:56.153 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-09-16T05:19:56.314 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T05:19:56.425 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T05:19:56.940 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T05:19:57.036 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T05:19:57.108 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T05:19:57.146 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 6/6 2024-09-16T05:19:57.872 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 6/6 2024-09-16T05:19:57.872 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/6 2024-09-16T05:19:57.873 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T05:19:57.873 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T05:19:57.873 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T05:19:57.873 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T05:19:58.142 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T05:19:58.142 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:19:58.142 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-09-16T05:19:58.142 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T05:19:58.142 INFO:teuthology.orchestra.run.smithi178.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T05:19:58.142 INFO:teuthology.orchestra.run.smithi178.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T05:19:58.143 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T05:19:58.143 INFO:teuthology.orchestra.run.smithi178.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T05:19:58.143 INFO:teuthology.orchestra.run.smithi178.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T05:19:58.143 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:19:58.143 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-09-16T05:19:58.396 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-fuse 2024-09-16T05:19:58.964 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:27 ago on Mon 16 Sep 2024 05:18:31 AM UTC. 2024-09-16T05:19:59.098 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-09-16T05:19:59.099 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:19:59.099 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-09-16T05:19:59.099 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:19:59.099 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-09-16T05:19:59.099 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 896 k 2024-09-16T05:19:59.099 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-09-16T05:19:59.099 INFO:teuthology.orchestra.run.smithi178.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T05:19:59.099 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:19:59.099 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-09-16T05:19:59.099 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:19:59.099 INFO:teuthology.orchestra.run.smithi178.stdout:Install 2 Packages 2024-09-16T05:19:59.099 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:19:59.100 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 976 k 2024-09-16T05:19:59.100 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 2.9 M 2024-09-16T05:19:59.100 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-09-16T05:19:59.267 INFO:teuthology.orchestra.run.smithi178.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 547 kB/s | 80 kB 00:00 2024-09-16T05:19:59.375 INFO:teuthology.orchestra.run.smithi178.stdout:(2/2): ceph-fuse-19.1.1-326.g8921c8ef.el9.x86_6 3.4 MB/s | 896 kB 00:00 2024-09-16T05:19:59.376 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:19:59.376 INFO:teuthology.orchestra.run.smithi178.stdout:Total 3.4 MB/s | 976 kB 00:00 2024-09-16T05:19:59.388 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-09-16T05:19:59.410 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-09-16T05:19:59.411 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-09-16T05:19:59.491 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-09-16T05:19:59.492 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-09-16T05:19:59.673 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-09-16T05:19:59.818 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T05:19:59.891 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2/2 2024-09-16T05:20:00.538 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2/2 2024-09-16T05:20:00.539 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/2 2024-09-16T05:20:00.826 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T05:20:00.826 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:20:00.826 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-09-16T05:20:00.827 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T05:20:00.827 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:20:00.827 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-09-16T05:20:01.147 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-volume 2024-09-16T05:20:01.731 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:30 ago on Mon 16 Sep 2024 05:18:31 AM UTC. 2024-09-16T05:20:01.867 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-09-16T05:20:01.868 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:20:01.868 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-09-16T05:20:01.868 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:20:01.869 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-09-16T05:20:01.869 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-volume noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 285 k 2024-09-16T05:20:01.869 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:20:01.869 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-09-16T05:20:01.869 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:20:01.869 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-09-16T05:20:01.870 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:20:01.870 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 285 k 2024-09-16T05:20:01.870 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 1.3 M 2024-09-16T05:20:01.870 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-09-16T05:20:02.105 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-volume-19.1.1-326.g8921c8ef.el9.noarch.rpm 1.2 MB/s | 285 kB 00:00 2024-09-16T05:20:02.106 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:20:02.106 INFO:teuthology.orchestra.run.smithi178.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-16T05:20:02.106 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-09-16T05:20:02.114 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-09-16T05:20:02.114 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-09-16T05:20:02.141 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-09-16T05:20:02.141 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-09-16T05:20:02.361 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-09-16T05:20:02.432 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T05:20:02.449 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T05:20:02.449 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T05:20:02.449 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T05:20:02.449 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:20:03.479 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T05:20:03.479 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:20:03.479 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-09-16T05:20:03.480 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T05:20:03.480 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:20:03.480 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-09-16T05:20:03.771 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install librados-devel 2024-09-16T05:20:04.354 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:33 ago on Mon 16 Sep 2024 05:18:31 AM UTC. 2024-09-16T05:20:04.492 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-09-16T05:20:04.493 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:20:04.493 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-09-16T05:20:04.493 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:20:04.493 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-09-16T05:20:04.493 INFO:teuthology.orchestra.run.smithi178.stdout: librados-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 127 k 2024-09-16T05:20:04.493 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:20:04.493 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-09-16T05:20:04.493 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:20:04.493 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-09-16T05:20:04.493 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:20:04.493 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 127 k 2024-09-16T05:20:04.494 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 456 k 2024-09-16T05:20:04.494 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-09-16T05:20:04.702 INFO:teuthology.orchestra.run.smithi178.stdout:librados-devel-19.1.1-326.g8921c8ef.el9.x86_64. 609 kB/s | 127 kB 00:00 2024-09-16T05:20:04.703 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:20:04.703 INFO:teuthology.orchestra.run.smithi178.stdout:Total 605 kB/s | 127 kB 00:00 2024-09-16T05:20:04.703 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-09-16T05:20:04.708 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-09-16T05:20:04.708 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-09-16T05:20:04.777 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-09-16T05:20:04.777 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-09-16T05:20:04.947 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-09-16T05:20:05.041 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T05:20:05.456 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T05:20:05.800 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T05:20:05.801 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:20:05.801 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-09-16T05:20:05.801 INFO:teuthology.orchestra.run.smithi178.stdout: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T05:20:05.801 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:20:05.801 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-09-16T05:20:06.018 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install libcephfs2 2024-09-16T05:20:06.594 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:35 ago on Mon 16 Sep 2024 05:18:31 AM UTC. 2024-09-16T05:20:06.689 INFO:teuthology.orchestra.run.smithi178.stdout:Package libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T05:20:06.734 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-09-16T05:20:06.735 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-09-16T05:20:06.735 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-09-16T05:20:06.806 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install libcephfs-devel 2024-09-16T05:20:07.373 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:36 ago on Mon 16 Sep 2024 05:18:31 AM UTC. 2024-09-16T05:20:07.505 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-09-16T05:20:07.506 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:20:07.506 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-09-16T05:20:07.506 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:20:07.507 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-09-16T05:20:07.507 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 32 k 2024-09-16T05:20:07.507 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:20:07.507 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-09-16T05:20:07.507 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:20:07.507 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-09-16T05:20:07.507 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:20:07.507 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 32 k 2024-09-16T05:20:07.507 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 142 k 2024-09-16T05:20:07.507 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-09-16T05:20:07.655 INFO:teuthology.orchestra.run.smithi178.stdout:libcephfs-devel-19.1.1-326.g8921c8ef.el9.x86_64 215 kB/s | 32 kB 00:00 2024-09-16T05:20:07.656 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:20:07.656 INFO:teuthology.orchestra.run.smithi178.stdout:Total 213 kB/s | 32 kB 00:00 2024-09-16T05:20:07.656 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-09-16T05:20:07.659 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-09-16T05:20:07.659 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-09-16T05:20:07.680 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-09-16T05:20:07.680 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-09-16T05:20:07.771 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-09-16T05:20:07.887 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T05:20:08.263 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T05:20:08.506 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T05:20:08.506 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:20:08.506 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-09-16T05:20:08.506 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T05:20:08.506 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:20:08.506 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-09-16T05:20:08.694 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install librados2 2024-09-16T05:20:09.262 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:38 ago on Mon 16 Sep 2024 05:18:31 AM UTC. 2024-09-16T05:20:09.352 INFO:teuthology.orchestra.run.smithi178.stdout:Package librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T05:20:09.396 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-09-16T05:20:09.397 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-09-16T05:20:09.397 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-09-16T05:20:09.464 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install librbd1 2024-09-16T05:20:10.029 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:39 ago on Mon 16 Sep 2024 05:18:31 AM UTC. 2024-09-16T05:20:10.119 INFO:teuthology.orchestra.run.smithi178.stdout:Package librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T05:20:10.162 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-09-16T05:20:10.163 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-09-16T05:20:10.164 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-09-16T05:20:10.228 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-rados 2024-09-16T05:20:10.793 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:39 ago on Mon 16 Sep 2024 05:18:31 AM UTC. 2024-09-16T05:20:10.883 INFO:teuthology.orchestra.run.smithi178.stdout:Package python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T05:20:10.927 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-09-16T05:20:10.928 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-09-16T05:20:10.929 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-09-16T05:20:10.999 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-rgw 2024-09-16T05:20:11.571 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:40 ago on Mon 16 Sep 2024 05:18:31 AM UTC. 2024-09-16T05:20:11.663 INFO:teuthology.orchestra.run.smithi178.stdout:Package python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T05:20:11.706 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-09-16T05:20:11.707 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-09-16T05:20:11.707 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-09-16T05:20:11.780 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-cephfs 2024-09-16T05:20:12.344 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:41 ago on Mon 16 Sep 2024 05:18:31 AM UTC. 2024-09-16T05:20:12.434 INFO:teuthology.orchestra.run.smithi178.stdout:Package python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T05:20:12.477 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-09-16T05:20:12.478 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-09-16T05:20:12.479 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-09-16T05:20:12.546 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-rbd 2024-09-16T05:20:13.113 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:42 ago on Mon 16 Sep 2024 05:18:31 AM UTC. 2024-09-16T05:20:13.203 INFO:teuthology.orchestra.run.smithi178.stdout:Package python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T05:20:13.245 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-09-16T05:20:13.246 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-09-16T05:20:13.246 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-09-16T05:20:13.321 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install rbd-fuse 2024-09-16T05:20:13.896 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:42 ago on Mon 16 Sep 2024 05:18:31 AM UTC. 2024-09-16T05:20:14.038 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-09-16T05:20:14.039 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:20:14.039 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2024-09-16T05:20:14.039 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:20:14.039 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-09-16T05:20:14.039 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 85 k 2024-09-16T05:20:14.039 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:20:14.040 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-09-16T05:20:14.040 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:20:14.040 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-09-16T05:20:14.040 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:20:14.040 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 85 k 2024-09-16T05:20:14.040 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 231 k 2024-09-16T05:20:14.040 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-09-16T05:20:14.233 INFO:teuthology.orchestra.run.smithi178.stdout:rbd-fuse-19.1.1-326.g8921c8ef.el9.x86_64.rpm 440 kB/s | 85 kB 00:00 2024-09-16T05:20:14.234 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:20:14.234 INFO:teuthology.orchestra.run.smithi178.stdout:Total 437 kB/s | 85 kB 00:00 2024-09-16T05:20:14.235 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-09-16T05:20:14.242 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-09-16T05:20:14.242 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-09-16T05:20:14.308 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-09-16T05:20:14.308 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-09-16T05:20:14.453 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-09-16T05:20:14.575 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T05:20:15.000 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T05:20:15.281 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T05:20:15.282 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:20:15.282 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-09-16T05:20:15.282 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T05:20:15.282 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:20:15.282 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-09-16T05:20:15.568 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install rbd-mirror 2024-09-16T05:20:16.152 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:45 ago on Mon 16 Sep 2024 05:18:31 AM UTC. 2024-09-16T05:20:16.288 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-09-16T05:20:16.289 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:20:16.289 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-09-16T05:20:16.289 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:20:16.289 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-09-16T05:20:16.289 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.1 M 2024-09-16T05:20:16.289 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:20:16.289 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-09-16T05:20:16.289 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:20:16.290 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-09-16T05:20:16.290 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:20:16.290 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 3.1 M 2024-09-16T05:20:16.290 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 13 M 2024-09-16T05:20:16.290 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-09-16T05:20:16.612 INFO:teuthology.orchestra.run.smithi178.stdout:rbd-mirror-19.1.1-326.g8921c8ef.el9.x86_64.rpm 9.7 MB/s | 3.1 MB 00:00 2024-09-16T05:20:16.612 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:20:16.613 INFO:teuthology.orchestra.run.smithi178.stdout:Total 9.6 MB/s | 3.1 MB 00:00 2024-09-16T05:20:16.613 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-09-16T05:20:16.621 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-09-16T05:20:16.621 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-09-16T05:20:16.702 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-09-16T05:20:16.702 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-09-16T05:20:17.138 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-09-16T05:20:17.187 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T05:20:17.213 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T05:20:17.213 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T05:20:17.213 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T05:20:17.213 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T05:20:17.213 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T05:20:17.213 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:20:18.370 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T05:20:18.370 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:20:18.370 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-09-16T05:20:18.370 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T05:20:18.370 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:20:18.370 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-09-16T05:20:18.610 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install rbd-nbd 2024-09-16T05:20:19.195 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:48 ago on Mon 16 Sep 2024 05:18:31 AM UTC. 2024-09-16T05:20:19.333 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-09-16T05:20:19.334 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:20:19.334 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2024-09-16T05:20:19.334 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:20:19.334 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-09-16T05:20:19.335 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-nbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 170 k 2024-09-16T05:20:19.335 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:20:19.335 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-09-16T05:20:19.335 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-09-16T05:20:19.335 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-09-16T05:20:19.335 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:20:19.335 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 170 k 2024-09-16T05:20:19.335 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 494 k 2024-09-16T05:20:19.335 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-09-16T05:20:19.556 INFO:teuthology.orchestra.run.smithi178.stdout:rbd-nbd-19.1.1-326.g8921c8ef.el9.x86_64.rpm 771 kB/s | 170 kB 00:00 2024-09-16T05:20:19.557 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-09-16T05:20:19.557 INFO:teuthology.orchestra.run.smithi178.stdout:Total 766 kB/s | 170 kB 00:00 2024-09-16T05:20:19.557 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-09-16T05:20:19.565 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-09-16T05:20:19.565 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-09-16T05:20:19.629 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-09-16T05:20:19.629 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-09-16T05:20:19.789 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-09-16T05:20:19.896 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T05:20:20.318 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T05:20:20.582 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T05:20:20.582 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:20:20.582 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-09-16T05:20:20.582 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T05:20:20.582 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-09-16T05:20:20.582 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-09-16T05:20:20.828 DEBUG:teuthology.parallel:result is None 2024-09-16T05:20:20.829 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T05:20:20.977 DEBUG:teuthology.orchestra.run.smithi178:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T05:20:20.999 INFO:teuthology.orchestra.run.smithi178.stdout:19.1.1-326.g8921c8ef.el9 2024-09-16T05:20:21.000 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326.g8921c8ef.el9 2024-09-16T05:20:21.000 INFO:teuthology.task.install:The correct ceph version 19.1.1-326.g8921c8ef is installed. 2024-09-16T05:20:21.002 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T05:20:21.002 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-09-16T05:20:21.002 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T05:20:21.066 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T05:20:21.066 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-09-16T05:20:21.066 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T05:20:21.130 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T05:20:21.195 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T05:20:21.195 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-09-16T05:20:21.196 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T05:20:21.260 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T05:20:21.325 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T05:20:21.325 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-09-16T05:20:21.326 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T05:20:21.390 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T05:20:21.454 INFO:teuthology.run_tasks:Running task workunit... 2024-09-16T05:20:21.471 INFO:tasks.workunit:Pulling workunits from ref ad35db06e8924912f01e32b35576297c0f1352b4 2024-09-16T05:20:21.472 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-16T05:20:21.472 INFO:tasks.workunit:timeout=3h 2024-09-16T05:20:21.472 INFO:tasks.workunit:cleanup=True 2024-09-16T05:20:21.472 DEBUG:teuthology.orchestra.run.smithi178:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-16T05:20:21.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T05:20:21.509 INFO:teuthology.orchestra.run.smithi178.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T05:20:21.510 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-16T05:20:21.564 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-16T05:20:21.565 DEBUG:teuthology.orchestra.run.smithi178:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-16T05:20:21.622 DEBUG:teuthology.orchestra.run.smithi178:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout ad35db06e8924912f01e32b35576297c0f1352b4 2024-09-16T05:20:21.679 INFO:tasks.workunit.client.0.smithi178.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-16T05:21:17.901 INFO:tasks.workunit.client.0.smithi178.stderr:Note: switching to 'ad35db06e8924912f01e32b35576297c0f1352b4'. 2024-09-16T05:21:17.902 INFO:tasks.workunit.client.0.smithi178.stderr: 2024-09-16T05:21:17.902 INFO:tasks.workunit.client.0.smithi178.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-16T05:21:17.902 INFO:tasks.workunit.client.0.smithi178.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-16T05:21:17.902 INFO:tasks.workunit.client.0.smithi178.stderr:state without impacting any branches by switching back to a branch. 2024-09-16T05:21:17.902 INFO:tasks.workunit.client.0.smithi178.stderr: 2024-09-16T05:21:17.902 INFO:tasks.workunit.client.0.smithi178.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-16T05:21:17.902 INFO:tasks.workunit.client.0.smithi178.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-16T05:21:17.902 INFO:tasks.workunit.client.0.smithi178.stderr: 2024-09-16T05:21:17.902 INFO:tasks.workunit.client.0.smithi178.stderr: git switch -c 2024-09-16T05:21:17.902 INFO:tasks.workunit.client.0.smithi178.stderr: 2024-09-16T05:21:17.902 INFO:tasks.workunit.client.0.smithi178.stderr:Or undo this operation with: 2024-09-16T05:21:17.903 INFO:tasks.workunit.client.0.smithi178.stderr: 2024-09-16T05:21:17.903 INFO:tasks.workunit.client.0.smithi178.stderr: git switch - 2024-09-16T05:21:17.903 INFO:tasks.workunit.client.0.smithi178.stderr: 2024-09-16T05:21:17.903 INFO:tasks.workunit.client.0.smithi178.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-16T05:21:17.903 INFO:tasks.workunit.client.0.smithi178.stderr: 2024-09-16T05:21:17.903 INFO:tasks.workunit.client.0.smithi178.stderr:HEAD is now at ad35db06e89 Merge pull request #59794 from zdover23/wip-doc-2024-09-15-backport-59613-to-squid 2024-09-16T05:21:17.905 DEBUG:teuthology.orchestra.run.smithi178:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-16T05:21:17.964 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-09-16T05:21:17.964 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-16T05:21:18.031 INFO:tasks.workunit:Running workunits matching misc on client.0... 2024-09-16T05:21:18.031 INFO:tasks.workunit:Running workunit misc/mclock-config.sh... 2024-09-16T05:21:18.032 DEBUG:teuthology.orchestra.run.smithi178:workunit test misc/mclock-config.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ad35db06e8924912f01e32b35576297c0f1352b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh 2024-09-16T05:21:18.119 INFO:tasks.workunit.client.0.smithi178.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-16T05:21:18.125 INFO:tasks.workunit.client.0.smithi178.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-16T05:21:18.125 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-16T05:21:18.125 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-16T05:21:18.125 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-09-16T05:21:18.125 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-09-16T05:21:18.125 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-09-16T05:21:18.125 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-09-16T05:21:18.126 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-09-16T05:21:18.126 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-09-16T05:21:18.126 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/mclock-config 2024-09-16T05:21:18.126 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:21: run: local dir=td/mclock-config 2024-09-16T05:21:18.126 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:22: run: shift 2024-09-16T05:21:18.126 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:24: run: export CEPH_MON=127.0.0.1:7124 2024-09-16T05:21:18.126 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:24: run: CEPH_MON=127.0.0.1:7124 2024-09-16T05:21:18.126 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:25: run: export CEPH_ARGS 2024-09-16T05:21:18.126 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:26: run: uuidgen 2024-09-16T05:21:18.133 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:26: run: CEPH_ARGS+='--fsid=64e3af7b-5a84-4040-b122-b03cebe716a6 --auth-supported=none ' 2024-09-16T05:21:18.133 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2024-09-16T05:21:18.133 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:28: run: CEPH_ARGS+='--debug-mclock 20 ' 2024-09-16T05:21:18.133 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: set 2024-09-16T05:21:18.133 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-16T05:21:18.135 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: local 'funcs=TEST_backfill_limit_adjustment_mclock 2024-09-16T05:21:18.135 INFO:tasks.workunit.client.0.smithi178.stderr:TEST_profile_builtin_to_custom 2024-09-16T05:21:18.135 INFO:tasks.workunit.client.0.smithi178.stderr:TEST_profile_custom_to_builtin 2024-09-16T05:21:18.136 INFO:tasks.workunit.client.0.smithi178.stderr:TEST_profile_disallow_builtin_params_modify 2024-09-16T05:21:18.136 INFO:tasks.workunit.client.0.smithi178.stderr:TEST_profile_disallow_builtin_params_override 2024-09-16T05:21:18.136 INFO:tasks.workunit.client.0.smithi178.stderr:TEST_recovery_limit_adjustment_mclock' 2024-09-16T05:21:18.136 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-09-16T05:21:18.136 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-09-16T05:21:18.136 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-16T05:21:18.136 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-16T05:21:18.136 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T05:21:18.136 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:21:18.136 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T05:21:18.136 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:21:18.136 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:21:18.137 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:21:18.137 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:21:18.137 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:21:18.137 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:21:18.137 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:21:18.138 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:21:18.138 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:21:18.138 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:21:18.138 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:21:18.138 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:21:18.139 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:21:18.139 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:21:18.140 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:21:18.140 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:21:18.140 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:21:18.141 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:21:18.141 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T05:21:18.143 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:21:18.143 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:21:18.143 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:21:18.143 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26942 2024-09-16T05:21:18.144 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:21:18.144 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:21:18.144 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-16T05:21:18.145 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:21:18.145 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:21:18.145 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:21:18.145 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26942 2024-09-16T05:21:18.148 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:21:18.148 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-16T05:21:18.148 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-16T05:21:18.148 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:21:18.148 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-16T05:21:18.148 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_backfill_limit_adjustment_mclock td/mclock-config 2024-09-16T05:21:18.148 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:236: TEST_backfill_limit_adjustment_mclock: local dir=td/mclock-config 2024-09-16T05:21:18.148 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:238: TEST_backfill_limit_adjustment_mclock: setup td/mclock-config 2024-09-16T05:21:18.148 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-16T05:21:18.148 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-16T05:21:18.149 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T05:21:18.149 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:21:18.149 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T05:21:18.149 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:21:18.149 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:21:18.149 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:21:18.149 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:21:18.149 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:21:18.150 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:21:18.150 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:21:18.150 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:21:18.150 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:21:18.150 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:21:18.151 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:21:18.151 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:21:18.151 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:21:18.152 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:21:18.152 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:21:18.152 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:21:18.153 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:21:18.153 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:21:18.153 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T05:21:18.155 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:21:18.156 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:21:18.156 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:21:18.156 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26942 2024-09-16T05:21:18.156 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:21:18.156 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:21:18.156 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-16T05:21:18.157 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:21:18.157 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:21:18.157 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:21:18.157 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26942 2024-09-16T05:21:18.159 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:21:18.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:21:18.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:21:18.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-16T05:21:18.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:239: TEST_backfill_limit_adjustment_mclock: run_mon td/mclock-config a 2024-09-16T05:21:18.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-09-16T05:21:18.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:21:18.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:21:18.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:21:18.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-09-16T05:21:18.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-09-16T05:21:18.622 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:21:18.623 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:21:18.623 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:21:18.623 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:21:18.623 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:21:18.623 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:21:18.623 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26942/$cluster-$name.asok' 2024-09-16T05:21:18.623 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26942/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T05:21:18.807 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:21:18.808 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:21:18.808 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:21:18.809 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:21:18.809 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:21:18.809 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:21:18.809 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:21:18.809 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:21:18.809 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:21:18.809 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:21:18.809 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:21:18.809 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:21:18.809 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-mon.a.asok 2024-09-16T05:21:18.809 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:21:18.809 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26942/ceph-mon.a.asok config get fsid 2024-09-16T05:21:18.890 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:21:18.890 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:21:18.890 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:21:18.890 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:21:18.890 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:21:18.890 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:21:18.890 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:21:18.891 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:21:18.891 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:21:18.891 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:21:18.891 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:21:18.891 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-mon.a.asok 2024-09-16T05:21:18.891 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:21:18.891 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26942/ceph-mon.a.asok config get mon_host 2024-09-16T05:21:18.968 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:240: TEST_backfill_limit_adjustment_mclock: run_mgr td/mclock-config x 2024-09-16T05:21:18.968 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-09-16T05:21:18.968 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T05:21:18.968 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T05:21:18.969 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T05:21:18.969 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-09-16T05:21:18.969 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T05:21:19.812 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T05:21:19.812 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:21:19.812 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:21:19.812 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:21:19.813 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:21:19.813 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:21:19.813 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26942/$cluster-$name.asok' 2024-09-16T05:21:19.813 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:21:19.813 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26942/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:21:19.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:242: TEST_backfill_limit_adjustment_mclock: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-09-16T05:21:19.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-09-16T05:21:19.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:21:19.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T05:21:19.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:21:19.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-09-16T05:21:19.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=64e3af7b-5a84-4040-b122-b03cebe716a6 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-16T05:21:19.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:21:19.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:21:19.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:21:19.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-16T05:21:19.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-16T05:21:19.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:21:19.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:21:19.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-16T05:21:19.843 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:21:19.844 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:21:19.844 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:21:19.844 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:21:19.844 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:21:19.844 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:21:19.844 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26942/$cluster-$name.asok' 2024-09-16T05:21:19.844 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26942/$cluster-$name.asok' 2024-09-16T05:21:19.844 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:21:19.844 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:21:19.844 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:21:19.845 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-16T05:21:19.845 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-16T05:21:19.845 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:21:19.845 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:21:19.845 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:21:19.845 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:21:19.845 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:21:19.845 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-16T05:21:19.845 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-09-16T05:21:19.845 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:21:19.846 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9bc20e2a-9af9-46dc-bb55-90d4bf0af037 2024-09-16T05:21:19.846 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 9bc20e2a-9af9-46dc-bb55-90d4bf0af037 2024-09-16T05:21:19.847 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9bc20e2a-9af9-46dc-bb55-90d4bf0af037' 2024-09-16T05:21:19.848 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:21:19.860 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBPwOdm0As6MxAA9AIqJtHlakRLh1UgE7qREw== 2024-09-16T05:21:19.860 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBPwOdm0As6MxAA9AIqJtHlakRLh1UgE7qREw=="}' 2024-09-16T05:21:19.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9bc20e2a-9af9-46dc-bb55-90d4bf0af037 -i td/mclock-config/0/new.json 2024-09-16T05:21:20.142 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:21:20.151 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-09-16T05:21:20.153 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=64e3af7b-5a84-4040-b122-b03cebe716a6 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26942/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQBPwOdm0As6MxAA9AIqJtHlakRLh1UgE7qREw== --osd-uuid 9bc20e2a-9af9-46dc-bb55-90d4bf0af037 2024-09-16T05:21:20.214 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:21:20.213+0000 7f7223064640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:21:20.219 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:21:20.218+0000 7f7223064640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:21:20.220 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:21:20.219+0000 7f7223064640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:21:20.221 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:21:20.220+0000 7f7223064640 -1 bdev(0x560c51705c00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2024-09-16T05:21:20.221 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:21:20.220+0000 7f7223064640 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-09-16T05:21:25.681 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-09-16T05:21:25.682 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:21:25.682 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T05:21:25.682 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T05:21:25.682 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:21:26.181 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:21:26.182 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T05:21:26.182 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=64e3af7b-5a84-4040-b122-b03cebe716a6 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26942/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-16T05:21:26.182 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:21:26.182 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:21:26.182 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:21:26.207 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:21:26.205+0000 7f874360d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:21:26.227 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:21:26.225+0000 7f874360d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:21:26.228 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:21:26.227+0000 7f874360d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:21:26.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T05:21:26.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:21:26.633 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:21:26.633 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:21:26.633 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:21:26.633 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:21:26.633 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:21:26.634 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:21:26.634 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:21:26.634 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:21:27.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:21:27.311 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:21:27.310+0000 7f874360d640 -1 Falling back to public interface 2024-09-16T05:21:28.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:21:28.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:21:28.043 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:21:28.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:21:28.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:21:28.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:21:28.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:21:29.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:21:29.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:21:29.419 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:21:29.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:21:29.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:21:29.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:21:29.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:21:30.141 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:21:30.140+0000 7f874360d640 -1 osd.0 0 log_to_monitors true 2024-09-16T05:21:30.796 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:21:30.796 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:21:30.796 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:21:30.796 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:21:30.797 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:21:30.797 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:21:31.235 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:21:32.237 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:21:32.237 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:21:32.237 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:21:32.238 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:21:32.238 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:21:32.238 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:21:32.646 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:21:33.648 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:21:33.648 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:21:33.648 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:21:33.649 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:21:33.649 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:21:33.649 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:21:34.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:21:35.040 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:21:35.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:21:35.041 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:21:35.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:21:35.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:21:35.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:21:35.458 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:21:36.459 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:21:36.459 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:21:36.459 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:21:36.460 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:21:36.460 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:21:36.460 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:21:36.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:21:37.883 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:21:37.883 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:21:37.883 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:21:37.883 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:21:37.883 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:21:37.884 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:21:38.274 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:21:39.276 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:21:39.276 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:21:39.277 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:21:39.277 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:21:39.277 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:21:39.277 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:21:39.656 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:21:40.657 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:21:40.658 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:21:40.658 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:21:40.658 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:21:40.658 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:21:40.658 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:21:41.062 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:21:42.063 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:21:42.063 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:21:42.063 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:21:42.064 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:21:42.064 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:21:42.064 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:21:42.501 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:21:43.502 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:21:43.502 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:21:43.502 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:21:43.503 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:21:43.503 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:21:43.503 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:21:44.064 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:21:45.066 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:21:45.066 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:21:45.066 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:21:45.066 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:21:45.067 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:21:45.067 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:21:45.517 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:21:46.518 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:21:46.518 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:21:46.518 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:21:46.519 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:21:46.519 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:21:46.519 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:21:47.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:21:48.218 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:21:48.219 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:21:48.219 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:21:48.219 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:21:48.220 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:21:48.220 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:21:49.065 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:21:50.065 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:21:50.066 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:21:50.066 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:21:50.066 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:21:50.067 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:21:50.067 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:21:51.073 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:21:52.074 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:21:52.074 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:21:52.074 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:21:52.075 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:21:52.075 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:21:52.075 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:21:53.348 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:21:54.350 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:21:54.350 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:21:54.351 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T05:21:54.351 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T05:21:54.351 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:21:54.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:21:55.213 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:21:56.214 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:21:56.214 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:21:56.215 INFO:tasks.workunit.client.0.smithi178.stdout:19 2024-09-16T05:21:56.215 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-16T05:21:56.215 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:21:56.215 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:21:56.592 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2212584323,v1:127.0.0.1:6803/2212584323] [v2:127.0.0.1:6804/2212584323,v1:127.0.0.1:6805/2212584323] exists,up 9bc20e2a-9af9-46dc-bb55-90d4bf0af037 2024-09-16T05:21:56.595 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:21:56.596 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:21:56.596 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:21:56.596 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-09-16T05:21:56.596 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: bc 2024-09-16T05:21:56.596 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-16T05:21:56.597 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:21:56.597 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:21:56.597 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:21:56.597 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:21:56.598 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:21:56.598 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-osd.0.asok 2024-09-16T05:21:56.598 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-16T05:21:56.598 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26942/ceph-osd.0.asok config get osd_max_backfills 2024-09-16T05:21:56.678 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: local backfills=1 2024-09-16T05:21:56.678 INFO:tasks.workunit.client.0.smithi178.stdout:osd_max_backfills: 1 2024-09-16T05:21:56.679 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:246: TEST_backfill_limit_adjustment_mclock: echo 'osd_max_backfills: 1' 2024-09-16T05:21:56.679 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:251: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_max_backfills 20 2024-09-16T05:21:57.256 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:252: TEST_backfill_limit_adjustment_mclock: sleep 2 2024-09-16T05:21:59.260 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-09-16T05:21:59.261 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: bc 2024-09-16T05:21:59.261 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-16T05:21:59.261 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:21:59.261 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:21:59.261 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:21:59.261 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:21:59.262 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:21:59.262 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-osd.0.asok 2024-09-16T05:21:59.262 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-16T05:21:59.262 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26942/ceph-osd.0.asok config get osd_max_backfills 2024-09-16T05:21:59.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: local max_backfills=1 2024-09-16T05:21:59.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:255: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2024-09-16T05:21:59.353 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2024-09-16T05:21:59.353 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: bc 2024-09-16T05:21:59.355 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-16T05:21:59.355 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:21:59.355 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:21:59.355 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:21:59.356 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:21:59.356 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:21:59.356 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-osd.0.asok 2024-09-16T05:21:59.356 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-16T05:21:59.356 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26942/ceph-osd.0.asok dump_recovery_reservations 2024-09-16T05:21:59.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: max_backfills=1 2024-09-16T05:21:59.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:260: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2024-09-16T05:21:59.445 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2024-09-16T05:21:59.445 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: bc 2024-09-16T05:21:59.445 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-16T05:21:59.445 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:21:59.445 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:21:59.445 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:21:59.445 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:21:59.445 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:21:59.445 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-osd.0.asok 2024-09-16T05:21:59.445 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-16T05:21:59.446 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26942/ceph-osd.0.asok dump_recovery_reservations 2024-09-16T05:21:59.531 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: max_backfills=1 2024-09-16T05:21:59.531 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:263: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2024-09-16T05:21:59.531 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:267: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_mclock_override_recovery_settings true 2024-09-16T05:21:59.976 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:268: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_max_backfills 20 2024-09-16T05:22:00.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:269: TEST_backfill_limit_adjustment_mclock: sleep 2 2024-09-16T05:22:02.421 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-09-16T05:22:02.421 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: bc 2024-09-16T05:22:02.421 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-16T05:22:02.421 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:22:02.421 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:22:02.422 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:22:02.422 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:22:02.422 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:22:02.422 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-osd.0.asok 2024-09-16T05:22:02.422 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-16T05:22:02.422 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26942/ceph-osd.0.asok config get osd_max_backfills 2024-09-16T05:22:02.517 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-09-16T05:22:02.517 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:272: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-09-16T05:22:02.518 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2024-09-16T05:22:02.518 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: bc 2024-09-16T05:22:02.518 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-16T05:22:02.518 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:22:02.518 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:22:02.518 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:22:02.518 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:22:02.519 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:22:02.519 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-osd.0.asok 2024-09-16T05:22:02.519 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-16T05:22:02.519 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26942/ceph-osd.0.asok dump_recovery_reservations 2024-09-16T05:22:02.607 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-09-16T05:22:02.607 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:277: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-09-16T05:22:02.607 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2024-09-16T05:22:02.607 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: bc 2024-09-16T05:22:02.608 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-16T05:22:02.609 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:22:02.609 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:22:02.609 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:22:02.609 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:22:02.609 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:22:02.609 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-osd.0.asok 2024-09-16T05:22:02.609 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-16T05:22:02.609 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26942/ceph-osd.0.asok dump_recovery_reservations 2024-09-16T05:22:02.695 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-09-16T05:22:02.695 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:280: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-09-16T05:22:02.695 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:284: TEST_backfill_limit_adjustment_mclock: kill_daemons td/mclock-config TERM osd 2024-09-16T05:22:02.695 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:22:02.695 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:22:02.696 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:22:02.696 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:22:02.696 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:22:03.001 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:22:03.001 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:285: TEST_backfill_limit_adjustment_mclock: ceph osd down 0 2024-09-16T05:22:03.429 INFO:tasks.workunit.client.0.smithi178.stderr:osd.0 is already down. 2024-09-16T05:22:03.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:286: TEST_backfill_limit_adjustment_mclock: wait_for_osd down 0 2024-09-16T05:22:03.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2024-09-16T05:22:03.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:22:03.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:22:03.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:22:03.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:22:03.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:22:03.442 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:22:03.442 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:22:03.442 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2024-09-16T05:22:03.849 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 7 last_clean_interval [0,0) [v2:127.0.0.1:6802/2212584323,v1:127.0.0.1:6803/2212584323] [v2:127.0.0.1:6804/2212584323,v1:127.0.0.1:6805/2212584323] exists 9bc20e2a-9af9-46dc-bb55-90d4bf0af037 2024-09-16T05:22:03.852 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:22:03.853 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:22:03.853 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:22:03.853 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:287: TEST_backfill_limit_adjustment_mclock: activate_osd td/mclock-config 0 --osd-op-queue=mclock_scheduler 2024-09-16T05:22:03.853 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/mclock-config 2024-09-16T05:22:03.853 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T05:22:03.853 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T05:22:03.853 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T05:22:03.853 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/mclock-config/0 2024-09-16T05:22:03.853 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=64e3af7b-5a84-4040-b122-b03cebe716a6 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-16T05:22:03.853 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:22:03.853 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:22:03.854 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:22:03.854 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-16T05:22:03.854 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-16T05:22:03.854 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T05:22:03.854 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T05:22:03.854 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-16T05:22:03.854 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T05:22:03.854 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:22:03.854 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:22:03.854 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:22:03.854 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:22:03.854 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:22:03.854 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26942/$cluster-$name.asok' 2024-09-16T05:22:03.855 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26942/$cluster-$name.asok' 2024-09-16T05:22:03.855 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:22:03.855 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-16T05:22:03.855 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-16T05:22:03.855 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:22:03.855 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:22:03.855 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:22:03.855 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:22:03.855 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T05:22:03.855 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=--osd-op-queue=mclock_scheduler 2024-09-16T05:22:03.855 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/mclock-config/0 2024-09-16T05:22:03.855 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T05:22:03.856 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:22:03.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=64e3af7b-5a84-4040-b122-b03cebe716a6 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26942/$cluster-$name.asok' --debug-osd=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler 2024-09-16T05:22:03.857 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/mclock-config/0/whoami 2024-09-16T05:22:03.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T05:22:03.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T05:22:03.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T05:22:03.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T05:22:03.879 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:22:03.878+0000 7f5a1d8e4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:22:03.880 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:22:03.880+0000 7f5a1d8e4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:22:03.881 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:22:03.881+0000 7f5a1d8e4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:22:04.260 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T05:22:04.260 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:22:04.261 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:22:04.261 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:22:04.261 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:22:04.261 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:22:04.261 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:22:04.262 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:22:04.262 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:22:04.262 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:22:04.633 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:22:05.234 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:22:05.233+0000 7f5a1d8e4640 -1 Falling back to public interface 2024-09-16T05:22:05.635 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:22:05.635 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:22:05.635 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:22:05.636 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:22:05.636 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:22:05.636 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:22:06.037 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:22:07.038 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:22:07.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:22:07.039 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:22:07.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:22:07.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:22:07.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:22:07.413 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:22:08.289 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:22:08.288+0000 7f5a1d8e4640 -1 osd.0 6 log_to_monitors true 2024-09-16T05:22:08.414 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:22:08.415 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:22:08.415 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:22:08.416 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:22:08.416 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:22:08.416 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:22:08.880 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/1424586458,v1:127.0.0.1:6803/1424586458] [v2:127.0.0.1:6804/1424586458,v1:127.0.0.1:6805/1424586458] exists,up 9bc20e2a-9af9-46dc-bb55-90d4bf0af037 2024-09-16T05:22:08.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:22:08.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:22:08.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:22:08.882 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-09-16T05:22:08.882 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: bc 2024-09-16T05:22:08.882 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-16T05:22:08.882 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:22:08.882 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:22:08.882 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:22:08.882 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:22:08.882 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:22:08.882 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-osd.0.asok 2024-09-16T05:22:08.882 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-16T05:22:08.882 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26942/ceph-osd.0.asok config get osd_max_backfills 2024-09-16T05:22:08.965 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-09-16T05:22:08.965 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:291: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-09-16T05:22:08.966 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2024-09-16T05:22:08.966 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: bc 2024-09-16T05:22:08.966 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-16T05:22:08.967 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:22:08.967 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:22:08.967 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:22:08.967 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:22:08.967 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:22:08.967 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-osd.0.asok 2024-09-16T05:22:08.967 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-16T05:22:08.967 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26942/ceph-osd.0.asok dump_recovery_reservations 2024-09-16T05:22:09.053 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-09-16T05:22:09.053 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:296: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-09-16T05:22:09.053 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2024-09-16T05:22:09.053 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: bc 2024-09-16T05:22:09.053 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-16T05:22:09.053 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:22:09.053 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:22:09.053 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:22:09.054 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:22:09.054 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:22:09.054 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-osd.0.asok 2024-09-16T05:22:09.054 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-16T05:22:09.054 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26942/ceph-osd.0.asok dump_recovery_reservations 2024-09-16T05:22:09.140 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-09-16T05:22:09.140 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:299: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-09-16T05:22:09.140 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:301: TEST_backfill_limit_adjustment_mclock: teardown td/mclock-config 2024-09-16T05:22:09.140 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T05:22:09.140 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:22:09.141 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T05:22:09.141 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:22:09.141 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:22:09.141 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:22:09.141 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:22:09.141 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:22:09.245 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:22:09.246 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:22:09.246 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:22:09.247 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:22:09.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:22:09.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:22:09.248 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:22:09.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:22:09.249 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:22:09.249 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:22:09.249 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:22:09.250 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:22:09.251 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:22:09.251 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T05:22:09.256 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:22:09.256 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:22:09.256 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:22:09.257 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26942 2024-09-16T05:22:09.258 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:22:09.258 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:22:09.258 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-09-16T05:22:09.258 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T05:22:09.258 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:22:09.258 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T05:22:09.258 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:22:09.258 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:22:09.259 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:22:09.259 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:22:09.259 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:22:09.259 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:22:09.259 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:22:09.260 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:22:09.260 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:22:09.262 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:22:09.262 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:22:09.262 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:22:09.263 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:22:09.263 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:22:09.263 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:22:09.263 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:22:09.263 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:22:09.264 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:22:09.265 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T05:22:09.266 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:22:09.266 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:22:09.266 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:22:09.266 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26942 2024-09-16T05:22:09.268 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:22:09.268 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:22:09.268 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-09-16T05:22:09.268 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-09-16T05:22:09.268 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-16T05:22:09.268 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-16T05:22:09.268 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T05:22:09.268 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:22:09.268 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T05:22:09.268 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:22:09.268 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:22:09.269 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:22:09.269 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:22:09.269 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:22:09.269 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:22:09.269 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:22:09.270 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:22:09.270 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:22:09.271 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:22:09.271 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:22:09.271 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:22:09.272 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:22:09.272 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:22:09.272 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:22:09.272 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:22:09.273 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:22:09.274 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:22:09.274 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T05:22:09.275 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:22:09.275 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:22:09.275 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:22:09.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26942 2024-09-16T05:22:09.276 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:22:09.276 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:22:09.276 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-16T05:22:09.277 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:22:09.277 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:22:09.277 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:22:09.277 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26942 2024-09-16T05:22:09.279 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:22:09.279 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:22:09.280 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:22:09.280 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-16T05:22:09.280 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_builtin_to_custom td/mclock-config 2024-09-16T05:22:09.280 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:39: TEST_profile_builtin_to_custom: local dir=td/mclock-config 2024-09-16T05:22:09.280 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:41: TEST_profile_builtin_to_custom: run_mon td/mclock-config a 2024-09-16T05:22:09.280 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-09-16T05:22:09.280 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:22:09.280 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:22:09.280 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:22:09.280 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-09-16T05:22:09.280 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-09-16T05:22:09.642 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:22:09.643 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:22:09.643 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:22:09.643 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:22:09.643 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:22:09.643 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:22:09.643 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26942/$cluster-$name.asok' 2024-09-16T05:22:09.643 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26942/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T05:22:09.827 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:22:09.829 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:22:09.829 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:22:09.829 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:22:09.829 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:22:09.829 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:22:09.829 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:22:09.829 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:22:09.829 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:22:09.829 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:22:09.829 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:22:09.829 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:22:09.829 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-mon.a.asok 2024-09-16T05:22:09.830 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:22:09.830 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26942/ceph-mon.a.asok config get fsid 2024-09-16T05:22:09.906 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:22:09.906 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:22:09.906 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:22:09.906 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:22:09.906 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:22:09.906 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:22:09.906 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:22:09.906 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:22:09.907 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:22:09.907 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:22:09.907 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:22:09.907 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-mon.a.asok 2024-09-16T05:22:09.907 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:22:09.907 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26942/ceph-mon.a.asok config get mon_host 2024-09-16T05:22:09.984 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:42: TEST_profile_builtin_to_custom: run_mgr td/mclock-config x 2024-09-16T05:22:09.984 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-09-16T05:22:09.984 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T05:22:09.984 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T05:22:09.984 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T05:22:09.984 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-09-16T05:22:09.984 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T05:22:10.821 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T05:22:10.821 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:22:10.822 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:22:10.822 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:22:10.822 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:22:10.822 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:22:10.822 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26942/$cluster-$name.asok' 2024-09-16T05:22:10.822 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:22:10.822 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26942/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:22:10.852 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:43: TEST_profile_builtin_to_custom: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-09-16T05:22:10.852 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-09-16T05:22:10.852 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:22:10.853 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T05:22:10.853 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:22:10.853 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-09-16T05:22:10.853 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=64e3af7b-5a84-4040-b122-b03cebe716a6 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-16T05:22:10.853 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:22:10.856 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:22:10.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:22:10.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-16T05:22:10.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-16T05:22:10.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:22:10.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:22:10.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-16T05:22:10.857 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:22:10.857 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:22:10.857 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:22:10.857 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:22:10.857 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:22:10.857 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:22:10.858 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26942/$cluster-$name.asok' 2024-09-16T05:22:10.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26942/$cluster-$name.asok' 2024-09-16T05:22:10.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:22:10.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:22:10.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:22:10.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-16T05:22:10.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-16T05:22:10.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:22:10.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:22:10.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:22:10.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:22:10.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:22:10.859 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-16T05:22:10.859 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-09-16T05:22:10.859 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:22:10.859 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a71c0035-fbbf-44dd-908d-749fed1cf5d1 2024-09-16T05:22:10.859 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a71c0035-fbbf-44dd-908d-749fed1cf5d1' 2024-09-16T05:22:10.860 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 a71c0035-fbbf-44dd-908d-749fed1cf5d1 2024-09-16T05:22:10.860 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:22:10.871 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCCwOdmOMndMxAASQKE2m3QXzaxIYQo1sxpQA== 2024-09-16T05:22:10.871 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCCwOdmOMndMxAASQKE2m3QXzaxIYQo1sxpQA=="}' 2024-09-16T05:22:10.871 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a71c0035-fbbf-44dd-908d-749fed1cf5d1 -i td/mclock-config/0/new.json 2024-09-16T05:22:11.151 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:22:11.160 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-09-16T05:22:11.161 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=64e3af7b-5a84-4040-b122-b03cebe716a6 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26942/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQCCwOdmOMndMxAASQKE2m3QXzaxIYQo1sxpQA== --osd-uuid a71c0035-fbbf-44dd-908d-749fed1cf5d1 2024-09-16T05:22:11.190 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:22:11.189+0000 7f4010c18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:22:11.220 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:22:11.219+0000 7f4010c18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:22:11.221 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:22:11.220+0000 7f4010c18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:22:11.221 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:22:11.220+0000 7f4010c18640 -1 bdev(0x5597f00cfc00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2024-09-16T05:22:11.221 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:22:11.221+0000 7f4010c18640 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-09-16T05:22:16.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-09-16T05:22:16.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:22:16.419 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T05:22:16.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T05:22:16.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:22:16.938 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T05:22:16.938 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:22:16.939 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=64e3af7b-5a84-4040-b122-b03cebe716a6 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26942/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-16T05:22:16.939 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:22:16.939 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:22:16.939 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:22:16.964 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:22:16.962+0000 7fa0ee41a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:22:16.966 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:22:16.964+0000 7fa0ee41a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:22:16.968 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:22:16.965+0000 7fa0ee41a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:22:17.310 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T05:22:17.310 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:22:17.310 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:22:17.310 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:22:17.311 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:22:17.311 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:22:17.311 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:22:17.311 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:22:17.311 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:22:17.311 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:22:17.715 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:22:18.307 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:22:18.305+0000 7fa0ee41a640 -1 Falling back to public interface 2024-09-16T05:22:18.716 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:22:18.716 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:22:18.717 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:22:18.717 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:22:18.717 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:22:18.717 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:22:19.090 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:22:20.092 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:22:20.092 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:22:20.093 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:22:20.094 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:22:20.094 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:22:20.094 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:22:20.476 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:22:20.923 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:22:20.921+0000 7fa0ee41a640 -1 osd.0 0 log_to_monitors true 2024-09-16T05:22:21.477 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:22:21.477 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:22:21.477 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:22:21.478 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:22:21.478 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:22:21.478 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:22:21.959 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:22:22.960 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:22:22.961 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:22:22.961 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:22:22.962 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:22:22.962 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:22:22.962 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:22:23.364 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:22:24.365 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:22:24.365 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:22:24.366 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:22:24.366 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:22:24.366 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:22:24.366 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:22:24.870 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:22:25.869 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:22:25.869 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:22:25.870 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:22:25.870 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:22:25.870 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:22:25.870 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:22:26.273 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:22:27.274 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:22:27.274 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:22:27.274 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:22:27.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:22:27.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:22:27.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:22:27.779 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:22:28.780 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:22:28.780 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:22:28.781 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:22:28.781 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:22:28.781 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:22:28.781 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:22:29.189 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:22:30.190 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:22:30.190 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:22:30.191 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:22:30.191 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:22:30.191 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:22:30.191 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:22:30.669 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:22:31.670 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:22:31.670 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:22:31.671 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:22:31.671 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:22:31.671 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:22:31.671 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:22:32.047 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:22:33.049 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:22:33.049 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:22:33.049 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:22:33.050 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:22:33.050 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:22:33.050 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:22:33.497 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:22:34.498 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:22:34.498 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:22:34.498 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:22:34.499 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:22:34.499 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:22:34.499 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:22:35.634 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:22:36.636 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:22:36.636 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:22:36.636 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:22:36.637 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:22:36.637 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:22:36.637 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:22:37.779 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:22:38.780 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:22:38.780 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:22:38.780 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:22:38.781 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:22:38.781 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:22:38.781 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:22:39.289 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:22:40.291 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:22:40.291 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:22:40.291 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:22:40.291 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:22:40.292 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:22:40.292 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:22:40.888 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:22:41.890 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:22:41.890 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:22:41.891 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:22:41.891 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:22:41.891 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:22:41.891 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:22:42.589 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:22:43.591 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:22:43.591 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:22:43.591 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:22:43.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:22:43.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:22:43.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:22:43.972 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2448843060,v1:127.0.0.1:6803/2448843060] [v2:127.0.0.1:6804/2448843060,v1:127.0.0.1:6805/2448843060] exists,up a71c0035-fbbf-44dd-908d-749fed1cf5d1 2024-09-16T05:22:43.972 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:22:43.973 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:22:43.973 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:22:43.973 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:46: TEST_profile_builtin_to_custom: ceph config get osd.0 osd_mclock_profile 2024-09-16T05:22:44.342 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:46: TEST_profile_builtin_to_custom: local mclock_profile=balanced 2024-09-16T05:22:44.342 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:47: TEST_profile_builtin_to_custom: test balanced = balanced 2024-09-16T05:22:44.344 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: jq .osd_mclock_profile 2024-09-16T05:22:44.344 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-09-16T05:22:44.344 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:22:44.344 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:22:44.344 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:22:44.344 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:22:44.344 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:22:44.344 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-osd.0.asok 2024-09-16T05:22:44.344 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-09-16T05:22:44.344 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.26942/ceph-osd.0.asok config get osd_mclock_profile 2024-09-16T05:22:44.428 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: mclock_profile='"high_recovery_ops"' 2024-09-16T05:22:44.428 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: eval echo '"high_recovery_ops"' 2024-09-16T05:22:44.429 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: echo high_recovery_ops 2024-09-16T05:22:44.429 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: mclock_profile=high_recovery_ops 2024-09-16T05:22:44.429 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:54: TEST_profile_builtin_to_custom: test high_recovery_ops = high_recovery_ops 2024-09-16T05:22:44.429 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:57: TEST_profile_builtin_to_custom: ceph tell osd.0 config set osd_mclock_profile custom 2024-09-16T05:22:44.531 INFO:tasks.workunit.client.0.smithi178.stdout:{ 2024-09-16T05:22:44.531 INFO:tasks.workunit.client.0.smithi178.stdout: "success": "" 2024-09-16T05:22:44.531 INFO:tasks.workunit.client.0.smithi178.stdout:} 2024-09-16T05:22:44.541 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: jq .osd_mclock_profile 2024-09-16T05:22:44.541 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-09-16T05:22:44.541 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:22:44.541 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:22:44.541 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:22:44.541 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:22:44.541 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:22:44.541 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-osd.0.asok 2024-09-16T05:22:44.542 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-09-16T05:22:44.542 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.26942/ceph-osd.0.asok config get osd_mclock_profile 2024-09-16T05:22:44.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: mclock_profile='"custom"' 2024-09-16T05:22:44.624 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: eval echo '"custom"' 2024-09-16T05:22:44.624 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: echo custom 2024-09-16T05:22:44.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: mclock_profile=custom 2024-09-16T05:22:44.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:63: TEST_profile_builtin_to_custom: test custom = custom 2024-09-16T05:22:44.625 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: jq .osd_mclock_scheduler_client_res 2024-09-16T05:22:44.626 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: bc 2024-09-16T05:22:44.626 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-09-16T05:22:44.626 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:22:44.626 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:22:44.626 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:22:44.626 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:22:44.626 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:22:44.626 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-osd.0.asok 2024-09-16T05:22:44.626 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-09-16T05:22:44.626 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.26942/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-16T05:22:44.709 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: local client_res=0.300000 2024-09-16T05:22:44.709 INFO:tasks.workunit.client.0.smithi178.stdout:client_res = 0.300000 2024-09-16T05:22:44.710 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:69: TEST_profile_builtin_to_custom: echo 'client_res = 0.300000' 2024-09-16T05:22:44.710 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: echo '0.300000 + 0.1' 2024-09-16T05:22:44.710 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: bc -l 2024-09-16T05:22:44.710 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: local client_res_new=.400000 2024-09-16T05:22:44.710 INFO:tasks.workunit.client.0.smithi178.stdout:client_res_new = .400000 2024-09-16T05:22:44.711 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:71: TEST_profile_builtin_to_custom: echo 'client_res_new = .400000' 2024-09-16T05:22:44.711 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:72: TEST_profile_builtin_to_custom: ceph config set osd.0 osd_mclock_scheduler_client_res .400000 2024-09-16T05:22:45.216 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:77: TEST_profile_builtin_to_custom: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-09-16T05:22:45.581 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:77: TEST_profile_builtin_to_custom: local res=0.400000 2024-09-16T05:22:45.582 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: echo '0.400000 != .400000' 2024-09-16T05:22:45.582 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: bc -l 2024-09-16T05:22:45.583 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: (( 0 )) 2024-09-16T05:22:45.584 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: jq .osd_mclock_scheduler_client_res 2024-09-16T05:22:45.585 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-09-16T05:22:45.585 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: bc 2024-09-16T05:22:45.585 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:22:45.585 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:22:45.585 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:22:45.585 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:22:45.585 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:22:45.585 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-osd.0.asok 2024-09-16T05:22:45.585 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-09-16T05:22:45.585 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.26942/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-16T05:22:45.669 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: res=0.400000 2024-09-16T05:22:45.670 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: echo '0.400000 != .400000' 2024-09-16T05:22:45.670 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: bc -l 2024-09-16T05:22:45.671 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: (( 0 )) 2024-09-16T05:22:45.671 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:89: TEST_profile_builtin_to_custom: teardown td/mclock-config 2024-09-16T05:22:45.671 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T05:22:45.672 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:22:45.672 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T05:22:45.672 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:22:45.672 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:22:45.672 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:22:45.672 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:22:45.672 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:22:45.777 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:22:45.778 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:22:45.778 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:22:45.779 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:22:45.779 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:22:45.780 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:22:45.780 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:22:45.780 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:22:45.781 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:22:45.781 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:22:45.781 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:22:45.782 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:22:45.783 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:22:45.783 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T05:22:45.788 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:22:45.788 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:22:45.788 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:22:45.788 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26942 2024-09-16T05:22:45.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:22:45.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:22:45.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-09-16T05:22:45.790 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T05:22:45.790 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:22:45.790 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T05:22:45.790 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:22:45.790 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:22:45.790 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:22:45.790 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:22:45.790 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:22:45.791 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:22:45.791 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:22:45.792 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:22:45.792 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:22:45.793 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:22:45.793 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:22:45.793 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:22:45.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:22:45.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:22:45.795 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:22:45.795 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:22:45.795 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:22:45.796 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:22:45.796 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T05:22:45.797 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:22:45.798 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:22:45.798 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:22:45.798 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26942 2024-09-16T05:22:45.799 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:22:45.799 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:22:45.799 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-09-16T05:22:45.799 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-09-16T05:22:45.800 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-16T05:22:45.800 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-16T05:22:45.800 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T05:22:45.800 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:22:45.800 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T05:22:45.800 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:22:45.800 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:22:45.800 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:22:45.800 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:22:45.800 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:22:45.801 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:22:45.801 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:22:45.801 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:22:45.802 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:22:45.802 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:22:45.803 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:22:45.803 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:22:45.803 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:22:45.804 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:22:45.804 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:22:45.804 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:22:45.805 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:22:45.805 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:22:45.806 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T05:22:45.807 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:22:45.807 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:22:45.807 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:22:45.807 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26942 2024-09-16T05:22:45.808 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:22:45.808 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:22:45.808 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-16T05:22:45.809 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:22:45.809 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:22:45.809 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:22:45.809 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26942 2024-09-16T05:22:45.812 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:22:45.812 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:22:45.812 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:22:45.812 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-16T05:22:45.812 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_custom_to_builtin td/mclock-config 2024-09-16T05:22:45.812 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:93: TEST_profile_custom_to_builtin: local dir=td/mclock-config 2024-09-16T05:22:45.812 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:95: TEST_profile_custom_to_builtin: setup td/mclock-config 2024-09-16T05:22:45.812 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-16T05:22:45.812 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-16T05:22:45.812 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T05:22:45.812 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:22:45.813 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T05:22:45.813 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:22:45.813 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:22:45.813 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:22:45.813 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:22:45.813 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:22:45.814 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:22:45.814 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:22:45.814 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:22:45.814 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:22:45.815 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:22:45.815 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:22:45.815 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:22:45.815 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:22:45.816 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:22:45.816 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:22:45.816 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:22:45.817 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:22:45.817 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:22:45.818 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T05:22:45.819 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:22:45.819 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:22:45.819 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:22:45.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26942 2024-09-16T05:22:45.820 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:22:45.820 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:22:45.820 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-16T05:22:45.821 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:22:45.821 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:22:45.821 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:22:45.821 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26942 2024-09-16T05:22:45.823 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:22:45.823 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:22:45.823 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:22:45.823 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-16T05:22:45.823 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:96: TEST_profile_custom_to_builtin: run_mon td/mclock-config a 2024-09-16T05:22:45.823 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-09-16T05:22:45.823 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:22:45.823 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:22:45.823 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:22:45.824 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-09-16T05:22:45.824 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-09-16T05:22:46.209 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:22:46.209 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:22:46.209 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:22:46.209 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:22:46.209 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:22:46.209 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:22:46.209 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26942/$cluster-$name.asok' 2024-09-16T05:22:46.210 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26942/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T05:22:46.397 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:22:46.399 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:22:46.399 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:22:46.399 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:22:46.399 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:22:46.399 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:22:46.399 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:22:46.399 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:22:46.399 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:22:46.399 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:22:46.400 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:22:46.400 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:22:46.400 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-mon.a.asok 2024-09-16T05:22:46.400 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:22:46.400 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26942/ceph-mon.a.asok config get fsid 2024-09-16T05:22:46.487 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:22:46.487 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:22:46.487 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:22:46.487 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:22:46.487 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:22:46.487 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:22:46.488 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:22:46.488 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:22:46.488 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:22:46.488 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:22:46.488 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:22:46.488 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-mon.a.asok 2024-09-16T05:22:46.488 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:22:46.488 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26942/ceph-mon.a.asok config get mon_host 2024-09-16T05:22:46.566 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:97: TEST_profile_custom_to_builtin: run_mgr td/mclock-config x 2024-09-16T05:22:46.566 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-09-16T05:22:46.566 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T05:22:46.566 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T05:22:46.566 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T05:22:46.566 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-09-16T05:22:46.566 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T05:22:47.409 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T05:22:47.410 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:22:47.410 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:22:47.410 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:22:47.410 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:22:47.410 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:22:47.410 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26942/$cluster-$name.asok' 2024-09-16T05:22:47.410 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:22:47.411 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26942/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:22:47.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:98: TEST_profile_custom_to_builtin: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-09-16T05:22:47.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-09-16T05:22:47.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:22:47.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T05:22:47.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:22:47.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-09-16T05:22:47.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=64e3af7b-5a84-4040-b122-b03cebe716a6 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-16T05:22:47.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:22:47.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:22:47.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:22:47.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-16T05:22:47.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-16T05:22:47.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:22:47.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:22:47.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-16T05:22:47.441 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:22:47.441 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:22:47.441 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:22:47.442 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:22:47.442 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:22:47.442 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:22:47.442 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26942/$cluster-$name.asok' 2024-09-16T05:22:47.442 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26942/$cluster-$name.asok' 2024-09-16T05:22:47.442 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:22:47.442 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:22:47.442 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:22:47.442 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-16T05:22:47.442 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-16T05:22:47.442 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:22:47.442 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:22:47.442 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:22:47.442 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:22:47.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:22:47.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-16T05:22:47.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-09-16T05:22:47.443 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:22:47.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=617d0994-0af0-4884-96ba-98ab1ec5d2da 2024-09-16T05:22:47.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 617d0994-0af0-4884-96ba-98ab1ec5d2da' 2024-09-16T05:22:47.444 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 617d0994-0af0-4884-96ba-98ab1ec5d2da 2024-09-16T05:22:47.444 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:22:47.458 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCnwOdmgjBEGxAAGLfDfsaOh00zjSzHpHi/RQ== 2024-09-16T05:22:47.458 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCnwOdmgjBEGxAAGLfDfsaOh00zjSzHpHi/RQ=="}' 2024-09-16T05:22:47.459 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 617d0994-0af0-4884-96ba-98ab1ec5d2da -i td/mclock-config/0/new.json 2024-09-16T05:22:47.729 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:22:47.738 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-09-16T05:22:47.739 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=64e3af7b-5a84-4040-b122-b03cebe716a6 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26942/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQCnwOdmgjBEGxAAGLfDfsaOh00zjSzHpHi/RQ== --osd-uuid 617d0994-0af0-4884-96ba-98ab1ec5d2da 2024-09-16T05:22:47.768 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:22:47.767+0000 7f25f4758640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:22:47.789 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:22:47.788+0000 7f25f4758640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:22:47.790 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:22:47.789+0000 7f25f4758640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:22:47.791 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:22:47.790+0000 7f25f4758640 -1 bdev(0x55a1f8821c00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2024-09-16T05:22:47.791 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:22:47.790+0000 7f25f4758640 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-09-16T05:22:53.733 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-09-16T05:22:53.734 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:22:53.734 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T05:22:53.734 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T05:22:53.734 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:22:54.251 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:22:54.252 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T05:22:54.252 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=64e3af7b-5a84-4040-b122-b03cebe716a6 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26942/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-16T05:22:54.252 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:22:54.252 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:22:54.252 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:22:54.277 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:22:54.276+0000 7f35e862a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:22:54.308 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:22:54.307+0000 7f35e862a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:22:54.309 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:22:54.308+0000 7f35e862a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:22:54.654 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T05:22:54.654 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:22:54.654 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:22:54.654 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:22:54.654 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:22:54.654 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:22:54.654 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:22:54.655 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:22:54.655 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:22:54.656 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:22:55.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:22:55.643 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:22:55.642+0000 7f35e862a640 -1 Falling back to public interface 2024-09-16T05:22:56.037 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:22:56.037 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:22:56.037 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:22:56.038 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:22:56.038 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:22:56.038 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:22:56.415 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:22:57.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:22:57.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:22:57.417 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:22:57.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:22:57.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:22:57.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:22:57.831 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:22:58.377 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:22:58.376+0000 7f35e862a640 -1 osd.0 0 log_to_monitors true 2024-09-16T05:22:58.833 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:22:58.833 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:22:58.833 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:22:58.834 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:22:58.834 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:22:58.834 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:22:59.311 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:23:00.313 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:23:00.313 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:23:00.313 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:23:00.313 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:23:00.314 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:23:00.314 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:23:00.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:23:01.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:23:01.720 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:23:01.720 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:23:01.720 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:23:01.720 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:23:01.720 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:23:02.160 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:23:03.161 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:23:03.161 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:23:03.161 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:23:03.162 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:23:03.162 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:23:03.162 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:23:03.542 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:23:04.544 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:23:04.544 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:23:04.544 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:23:04.545 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:23:04.548 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:23:04.548 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:23:04.931 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:23:05.932 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:23:05.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:23:05.933 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:23:05.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:23:05.934 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:23:05.934 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:23:06.315 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:23:07.317 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:23:07.317 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:23:07.317 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:23:07.318 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:23:07.318 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:23:07.318 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:23:07.717 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:23:08.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:23:08.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:23:08.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:23:08.718 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:23:08.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:23:08.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:23:09.244 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:23:10.245 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:23:10.245 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:23:10.246 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:23:10.246 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:23:10.246 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:23:10.246 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:23:11.566 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:23:12.568 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:23:12.568 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:23:12.568 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:23:12.568 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:23:12.569 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:23:12.569 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:23:13.446 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:23:14.448 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:23:14.448 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:23:14.448 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:23:14.449 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:23:14.449 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:23:14.449 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:23:15.268 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:23:16.270 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:23:16.270 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:23:16.271 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:23:16.272 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:23:16.272 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:23:16.272 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:23:16.951 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:23:17.953 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:23:17.953 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:23:17.953 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:23:17.954 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:23:17.954 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:23:17.954 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:23:18.510 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:23:19.511 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:23:19.511 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:23:19.512 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:23:19.512 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:23:19.513 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:23:19.513 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:23:20.479 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:23:21.481 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:23:21.481 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:23:21.481 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:23:21.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:23:21.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:23:21.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:23:21.910 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4133175667,v1:127.0.0.1:6803/4133175667] [v2:127.0.0.1:6804/4133175667,v1:127.0.0.1:6805/4133175667] exists,up 617d0994-0af0-4884-96ba-98ab1ec5d2da 2024-09-16T05:23:21.911 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:23:21.911 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:23:21.911 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:23:21.912 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:101: TEST_profile_custom_to_builtin: local def_mclock_profile 2024-09-16T05:23:21.912 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:102: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_profile 2024-09-16T05:23:22.280 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:102: TEST_profile_custom_to_builtin: def_mclock_profile=balanced 2024-09-16T05:23:22.280 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:103: TEST_profile_custom_to_builtin: test balanced = balanced 2024-09-16T05:23:22.281 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2024-09-16T05:23:22.281 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-16T05:23:22.281 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:23:22.282 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:23:22.282 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:23:22.282 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:23:22.282 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:23:22.282 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-osd.0.asok 2024-09-16T05:23:22.282 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-16T05:23:22.282 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.26942/ceph-osd.0.asok config get osd_mclock_profile 2024-09-16T05:23:22.366 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: local 'orig_mclock_profile="high_recovery_ops"' 2024-09-16T05:23:22.366 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: eval echo '"high_recovery_ops"' 2024-09-16T05:23:22.367 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: echo high_recovery_ops 2024-09-16T05:23:22.367 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: orig_mclock_profile=high_recovery_ops 2024-09-16T05:23:22.367 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:110: TEST_profile_custom_to_builtin: test high_recovery_ops = high_recovery_ops 2024-09-16T05:23:22.367 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:113: TEST_profile_custom_to_builtin: ceph tell osd.0 config set osd_mclock_profile custom 2024-09-16T05:23:22.496 INFO:tasks.workunit.client.0.smithi178.stdout:{ 2024-09-16T05:23:22.496 INFO:tasks.workunit.client.0.smithi178.stdout: "success": "" 2024-09-16T05:23:22.496 INFO:tasks.workunit.client.0.smithi178.stdout:} 2024-09-16T05:23:22.503 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2024-09-16T05:23:22.503 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-16T05:23:22.503 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:23:22.503 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:23:22.504 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:23:22.504 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:23:22.504 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:23:22.504 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-osd.0.asok 2024-09-16T05:23:22.504 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-16T05:23:22.504 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.26942/ceph-osd.0.asok config get osd_mclock_profile 2024-09-16T05:23:22.587 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: local 'mclock_profile="custom"' 2024-09-16T05:23:22.587 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: eval echo '"custom"' 2024-09-16T05:23:22.587 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: echo custom 2024-09-16T05:23:22.587 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: mclock_profile=custom 2024-09-16T05:23:22.587 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:120: TEST_profile_custom_to_builtin: test custom = custom 2024-09-16T05:23:22.587 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:123: TEST_profile_custom_to_builtin: local client_res 2024-09-16T05:23:22.589 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-09-16T05:23:22.589 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: bc 2024-09-16T05:23:22.589 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-16T05:23:22.589 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:23:22.589 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:23:22.589 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:23:22.589 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:23:22.589 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:23:22.589 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-osd.0.asok 2024-09-16T05:23:22.589 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-16T05:23:22.589 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.26942/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-16T05:23:22.672 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: client_res=0.300000 2024-09-16T05:23:22.672 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:127: TEST_profile_custom_to_builtin: echo 'Original client_res for osd.0 = 0.300000' 2024-09-16T05:23:22.673 INFO:tasks.workunit.client.0.smithi178.stdout:Original client_res for osd.0 = 0.300000 2024-09-16T05:23:22.673 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: echo '0.300000 + 0.1' 2024-09-16T05:23:22.673 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: bc -l 2024-09-16T05:23:22.674 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: local client_res_new=.400000 2024-09-16T05:23:22.674 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:131: TEST_profile_custom_to_builtin: echo 'client_res_new = .400000' 2024-09-16T05:23:22.674 INFO:tasks.workunit.client.0.smithi178.stdout:client_res_new = .400000 2024-09-16T05:23:22.674 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:132: TEST_profile_custom_to_builtin: ceph config set osd osd_mclock_scheduler_client_res .400000 2024-09-16T05:23:23.112 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:136: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-09-16T05:23:23.547 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:136: TEST_profile_custom_to_builtin: local res=0.400000 2024-09-16T05:23:23.548 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-09-16T05:23:23.548 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: bc -l 2024-09-16T05:23:23.548 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: (( 0 )) 2024-09-16T05:23:23.550 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-09-16T05:23:23.550 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: bc 2024-09-16T05:23:23.550 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-16T05:23:23.550 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:23:23.550 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:23:23.550 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:23:23.550 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:23:23.551 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:23:23.551 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-osd.0.asok 2024-09-16T05:23:23.551 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-16T05:23:23.551 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.26942/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-16T05:23:23.634 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: res=0.400000 2024-09-16T05:23:23.634 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-09-16T05:23:23.634 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: bc -l 2024-09-16T05:23:23.635 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: (( 0 )) 2024-09-16T05:23:23.635 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:154: TEST_profile_custom_to_builtin: ceph tell osd.0 config set osd_mclock_profile high_recovery_ops 2024-09-16T05:23:23.769 INFO:tasks.workunit.client.0.smithi178.stdout:{ 2024-09-16T05:23:23.769 INFO:tasks.workunit.client.0.smithi178.stdout: "success": "" 2024-09-16T05:23:23.769 INFO:tasks.workunit.client.0.smithi178.stdout:} 2024-09-16T05:23:23.780 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2024-09-16T05:23:23.780 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-16T05:23:23.780 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:23:23.780 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:23:23.780 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:23:23.780 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:23:23.780 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:23:23.780 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-osd.0.asok 2024-09-16T05:23:23.780 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-16T05:23:23.780 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.26942/ceph-osd.0.asok config get osd_mclock_profile 2024-09-16T05:23:23.863 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: eval 'mclock_profile="high_recovery_ops"' 2024-09-16T05:23:23.863 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: mclock_profile=high_recovery_ops 2024-09-16T05:23:23.863 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:160: TEST_profile_custom_to_builtin: test high_recovery_ops = high_recovery_ops 2024-09-16T05:23:23.863 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:165: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-09-16T05:23:24.228 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:165: TEST_profile_custom_to_builtin: local res=0.400000 2024-09-16T05:23:24.229 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-09-16T05:23:24.229 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: bc -l 2024-09-16T05:23:24.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: (( 0 )) 2024-09-16T05:23:24.231 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-09-16T05:23:24.231 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: bc 2024-09-16T05:23:24.231 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-16T05:23:24.231 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:23:24.231 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:23:24.231 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:23:24.231 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:23:24.232 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:23:24.232 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-osd.0.asok 2024-09-16T05:23:24.232 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-16T05:23:24.232 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.26942/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-16T05:23:24.317 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: res=0.400000 2024-09-16T05:23:24.317 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-09-16T05:23:24.317 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: bc -l 2024-09-16T05:23:24.318 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: (( 0 )) 2024-09-16T05:23:24.318 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:178: TEST_profile_custom_to_builtin: ceph config rm osd osd_mclock_scheduler_client_res 2024-09-16T05:23:24.774 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:180: TEST_profile_custom_to_builtin: sleep 5 2024-09-16T05:23:29.776 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:185: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-09-16T05:23:30.151 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:185: TEST_profile_custom_to_builtin: res=0.000000 2024-09-16T05:23:30.152 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: echo '0.000000 != 0.0' 2024-09-16T05:23:30.152 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: bc -l 2024-09-16T05:23:30.153 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: (( 0 )) 2024-09-16T05:23:30.154 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-09-16T05:23:30.154 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: bc 2024-09-16T05:23:30.154 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-16T05:23:30.156 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:23:30.156 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:23:30.156 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:23:30.156 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:23:30.156 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:23:30.156 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-osd.0.asok 2024-09-16T05:23:30.156 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-16T05:23:30.156 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.26942/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-16T05:23:30.239 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: res=0.300000 2024-09-16T05:23:30.240 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: echo '0.300000 != 0.300000' 2024-09-16T05:23:30.240 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: bc -l 2024-09-16T05:23:30.242 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: (( 0 )) 2024-09-16T05:23:30.242 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:198: TEST_profile_custom_to_builtin: teardown td/mclock-config 2024-09-16T05:23:30.242 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T05:23:30.242 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:23:30.242 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T05:23:30.242 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:23:30.242 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:23:30.242 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:23:30.242 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:23:30.242 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:23:30.348 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:23:30.348 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:23:30.349 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:23:30.349 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:23:30.350 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:23:30.351 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:23:30.351 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:23:30.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:23:30.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:23:30.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:23:30.352 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:23:30.354 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:23:30.354 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:23:30.355 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T05:23:30.359 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:23:30.359 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:23:30.359 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:23:30.359 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26942 2024-09-16T05:23:30.360 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:23:30.360 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:23:30.360 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-09-16T05:23:30.360 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T05:23:30.361 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:23:30.361 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T05:23:30.361 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:23:30.361 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:23:30.361 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:23:30.361 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:23:30.361 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:23:30.362 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:23:30.362 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:23:30.364 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:23:30.364 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:23:30.365 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:23:30.365 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:23:30.365 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:23:30.365 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:23:30.365 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:23:30.366 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:23:30.366 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:23:30.366 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:23:30.367 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:23:30.367 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T05:23:30.368 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:23:30.368 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:23:30.369 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:23:30.369 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26942 2024-09-16T05:23:30.370 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:23:30.370 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:23:30.370 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-09-16T05:23:30.370 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-09-16T05:23:30.371 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-16T05:23:30.371 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-16T05:23:30.371 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T05:23:30.371 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:23:30.371 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T05:23:30.371 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:23:30.371 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:23:30.371 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:23:30.371 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:23:30.371 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:23:30.372 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:23:30.372 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:23:30.373 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:23:30.373 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:23:30.374 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:23:30.374 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:23:30.374 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:23:30.375 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:23:30.375 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:23:30.375 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:23:30.375 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:23:30.376 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:23:30.377 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:23:30.377 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T05:23:30.378 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:23:30.378 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:23:30.378 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:23:30.378 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26942 2024-09-16T05:23:30.379 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:23:30.379 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:23:30.379 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-16T05:23:30.381 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:23:30.381 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:23:30.381 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:23:30.381 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26942 2024-09-16T05:23:30.383 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:23:30.383 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:23:30.383 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:23:30.384 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-16T05:23:30.384 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_disallow_builtin_params_modify td/mclock-config 2024-09-16T05:23:30.384 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:305: TEST_profile_disallow_builtin_params_modify: local dir=td/mclock-config 2024-09-16T05:23:30.384 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:307: TEST_profile_disallow_builtin_params_modify: setup td/mclock-config 2024-09-16T05:23:30.384 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-16T05:23:30.384 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-16T05:23:30.384 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T05:23:30.384 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:23:30.384 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T05:23:30.384 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:23:30.384 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:23:30.384 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:23:30.384 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:23:30.385 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:23:30.385 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:23:30.385 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:23:30.385 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:23:30.386 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:23:30.386 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:23:30.387 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:23:30.387 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:23:30.387 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:23:30.387 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:23:30.388 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:23:30.388 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:23:30.388 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:23:30.389 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:23:30.390 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T05:23:30.391 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:23:30.391 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:23:30.391 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:23:30.391 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26942 2024-09-16T05:23:30.392 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:23:30.392 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:23:30.392 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-16T05:23:30.393 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:23:30.393 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:23:30.393 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:23:30.393 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26942 2024-09-16T05:23:30.395 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:23:30.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:23:30.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:23:30.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-16T05:23:30.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:308: TEST_profile_disallow_builtin_params_modify: run_mon td/mclock-config a 2024-09-16T05:23:30.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-09-16T05:23:30.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:23:30.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:23:30.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:23:30.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-09-16T05:23:30.396 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-09-16T05:23:30.773 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:23:30.773 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:23:30.773 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:23:30.773 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:23:30.774 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:23:30.774 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:23:30.774 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26942/$cluster-$name.asok' 2024-09-16T05:23:30.774 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26942/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T05:23:30.955 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:23:30.957 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:23:30.957 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:23:30.957 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:23:30.957 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:23:30.957 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:23:30.957 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:23:30.957 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:23:30.957 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:23:30.957 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:23:30.957 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:23:30.957 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:23:30.957 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-mon.a.asok 2024-09-16T05:23:30.958 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:23:30.958 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26942/ceph-mon.a.asok config get fsid 2024-09-16T05:23:31.036 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:23:31.036 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:23:31.036 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:23:31.036 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:23:31.036 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:23:31.036 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:23:31.036 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:23:31.036 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:23:31.037 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:23:31.037 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:23:31.037 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:23:31.037 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-mon.a.asok 2024-09-16T05:23:31.037 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:23:31.037 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26942/ceph-mon.a.asok config get mon_host 2024-09-16T05:23:31.113 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:309: TEST_profile_disallow_builtin_params_modify: run_mgr td/mclock-config x 2024-09-16T05:23:31.114 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-09-16T05:23:31.114 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T05:23:31.114 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T05:23:31.114 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T05:23:31.114 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-09-16T05:23:31.114 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T05:23:31.949 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T05:23:31.949 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:23:31.949 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:23:31.949 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:23:31.949 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:23:31.950 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:23:31.950 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26942/$cluster-$name.asok' 2024-09-16T05:23:31.950 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:23:31.950 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26942/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:23:31.980 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:311: TEST_profile_disallow_builtin_params_modify: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-09-16T05:23:31.980 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-09-16T05:23:31.981 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:23:31.981 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T05:23:31.981 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:23:31.981 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-09-16T05:23:31.981 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=64e3af7b-5a84-4040-b122-b03cebe716a6 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-16T05:23:31.981 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:23:31.981 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:23:31.981 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:23:31.981 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-16T05:23:31.981 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-16T05:23:31.981 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:23:31.981 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:23:31.981 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-16T05:23:31.982 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:23:31.982 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:23:31.982 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:23:31.982 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:23:31.982 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:23:31.982 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:23:31.982 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26942/$cluster-$name.asok' 2024-09-16T05:23:31.982 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26942/$cluster-$name.asok' 2024-09-16T05:23:31.982 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:23:31.982 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:23:31.982 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:23:31.982 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-16T05:23:31.982 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-16T05:23:31.983 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:23:31.983 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:23:31.983 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:23:31.983 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:23:31.983 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:23:31.983 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-16T05:23:31.983 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-09-16T05:23:31.983 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:23:31.983 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0bb79790-685d-4cac-8394-9abff8ea124a 2024-09-16T05:23:31.983 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0bb79790-685d-4cac-8394-9abff8ea124a' 2024-09-16T05:23:31.984 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 0bb79790-685d-4cac-8394-9abff8ea124a 2024-09-16T05:23:31.984 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:23:32.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDTwOdmeBprOxAAQ9pJjAyexAisBP3rVXTIaA== 2024-09-16T05:23:32.001 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDTwOdmeBprOxAAQ9pJjAyexAisBP3rVXTIaA=="}' 2024-09-16T05:23:32.001 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0bb79790-685d-4cac-8394-9abff8ea124a -i td/mclock-config/0/new.json 2024-09-16T05:23:32.271 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:23:32.280 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-09-16T05:23:32.281 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=64e3af7b-5a84-4040-b122-b03cebe716a6 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26942/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQDTwOdmeBprOxAAQ9pJjAyexAisBP3rVXTIaA== --osd-uuid 0bb79790-685d-4cac-8394-9abff8ea124a 2024-09-16T05:23:32.309 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:23:32.309+0000 7f458b2a5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:23:32.356 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:23:32.355+0000 7f458b2a5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:23:32.357 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:23:32.356+0000 7f458b2a5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:23:32.357 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:23:32.356+0000 7f458b2a5640 -1 bdev(0x5637e81afc00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2024-09-16T05:23:32.357 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:23:32.356+0000 7f458b2a5640 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-09-16T05:23:38.102 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-09-16T05:23:38.102 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:23:38.103 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T05:23:38.103 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T05:23:38.103 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:23:38.613 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T05:23:38.614 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:23:38.615 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=64e3af7b-5a84-4040-b122-b03cebe716a6 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26942/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-16T05:23:38.615 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:23:38.615 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:23:38.615 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:23:38.639 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:23:38.638+0000 7ff0c0fbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:23:38.666 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:23:38.665+0000 7ff0c0fbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:23:38.667 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:23:38.667+0000 7ff0c0fbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:23:39.022 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T05:23:39.022 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:23:39.022 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:23:39.022 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:23:39.022 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:23:39.022 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:23:39.023 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:23:39.023 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:23:39.023 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:23:39.023 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:23:39.438 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:23:40.264 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:23:40.263+0000 7ff0c0fbd640 -1 Falling back to public interface 2024-09-16T05:23:40.439 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:23:40.439 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:23:40.440 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:23:40.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:23:40.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:23:40.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:23:40.855 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:23:41.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:23:41.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:23:41.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:23:41.857 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:23:41.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:23:41.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:23:42.264 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:23:42.911 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:23:42.910+0000 7ff0c0fbd640 -1 osd.0 0 log_to_monitors true 2024-09-16T05:23:43.266 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:23:43.266 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:23:43.266 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:23:43.267 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:23:43.267 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:23:43.267 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:23:43.644 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:23:44.646 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:23:44.646 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:23:44.646 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:23:44.647 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:23:44.647 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:23:44.647 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:23:45.024 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:23:46.025 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:23:46.025 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:23:46.025 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:23:46.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:23:46.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:23:46.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:23:46.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:23:47.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:23:47.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:23:47.421 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:23:47.422 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:23:47.422 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:23:47.422 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:23:47.801 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:23:48.802 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:23:48.802 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:23:48.803 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:23:48.803 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:23:48.803 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:23:48.803 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:23:49.223 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:23:50.224 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:23:50.224 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:23:50.225 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:23:50.228 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:23:50.229 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:23:50.229 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:23:50.655 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:23:51.656 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:23:51.656 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:23:51.656 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:23:51.657 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:23:51.657 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:23:51.657 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:23:52.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:23:53.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:23:53.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:23:53.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:23:53.036 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:23:53.037 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:23:53.037 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:23:53.426 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:23:54.427 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:23:54.427 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:23:54.428 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:23:54.428 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:23:54.429 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:23:54.429 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:23:55.229 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:23:56.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:23:56.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:23:56.231 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:23:56.231 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:23:56.231 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:23:56.231 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:23:56.880 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:23:57.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:23:57.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:23:57.881 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:23:57.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:23:57.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:23:57.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:23:58.704 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:23:59.705 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:23:59.705 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:23:59.705 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:23:59.706 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:23:59.706 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:23:59.706 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:24:00.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:24:01.442 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:24:01.442 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:24:01.442 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:24:01.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:24:01.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:24:01.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:24:02.179 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:24:03.180 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:24:03.181 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:24:03.181 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:24:03.181 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:24:03.182 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:24:03.182 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:24:03.911 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:24:04.912 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:24:04.913 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:24:04.913 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:24:04.913 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:24:04.913 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:24:04.914 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:24:05.374 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:24:06.376 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:24:06.376 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:24:06.376 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T05:24:06.377 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T05:24:06.377 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:24:06.377 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:24:06.792 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3132717977,v1:127.0.0.1:6803/3132717977] [v2:127.0.0.1:6804/3132717977,v1:127.0.0.1:6805/3132717977] exists,up 0bb79790-685d-4cac-8394-9abff8ea124a 2024-09-16T05:24:06.792 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:24:06.793 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:24:06.793 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:24:06.793 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:314: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_profile 2024-09-16T05:24:07.197 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:314: TEST_profile_disallow_builtin_params_modify: local def_mclock_profile=balanced 2024-09-16T05:24:07.197 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:315: TEST_profile_disallow_builtin_params_modify: test balanced = balanced 2024-09-16T05:24:07.198 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_profile 2024-09-16T05:24:07.198 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-09-16T05:24:07.199 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:24:07.199 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:24:07.199 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:24:07.199 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:24:07.199 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:24:07.199 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-osd.0.asok 2024-09-16T05:24:07.199 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-09-16T05:24:07.199 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.26942/ceph-osd.0.asok config get osd_mclock_profile 2024-09-16T05:24:07.284 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: local 'cur_mclock_profile="high_recovery_ops"' 2024-09-16T05:24:07.285 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: eval echo '"high_recovery_ops"' 2024-09-16T05:24:07.285 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: echo high_recovery_ops 2024-09-16T05:24:07.285 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: cur_mclock_profile=high_recovery_ops 2024-09-16T05:24:07.285 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:322: TEST_profile_disallow_builtin_params_modify: test high_recovery_ops = high_recovery_ops 2024-09-16T05:24:07.285 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:324: TEST_profile_disallow_builtin_params_modify: options=('osd_mclock_scheduler_background_recovery_res' 'osd_mclock_scheduler_client_res') 2024-09-16T05:24:07.285 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:324: TEST_profile_disallow_builtin_params_modify: declare -a options 2024-09-16T05:24:07.285 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:327: TEST_profile_disallow_builtin_params_modify: local retries=10 2024-09-16T05:24:07.285 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:328: TEST_profile_disallow_builtin_params_modify: local errors=0 2024-09-16T05:24:07.285 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:329: TEST_profile_disallow_builtin_params_modify: for opt in "${options[@]}" 2024-09-16T05:24:07.285 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_background_recovery_res 2024-09-16T05:24:07.285 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: bc 2024-09-16T05:24:07.285 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-09-16T05:24:07.286 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:24:07.286 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:24:07.286 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:24:07.286 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:24:07.286 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:24:07.286 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-osd.0.asok 2024-09-16T05:24:07.286 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-09-16T05:24:07.286 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.26942/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-09-16T05:24:07.368 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: local opt_val_orig=0.700000 2024-09-16T05:24:07.368 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: echo '0.700000 + 0.1' 2024-09-16T05:24:07.368 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-16T05:24:07.369 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: local opt_val_new=.800000 2024-09-16T05:24:07.369 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:335: TEST_profile_disallow_builtin_params_modify: ceph config set osd.0 osd_mclock_scheduler_background_recovery_res .800000 2024-09-16T05:24:07.923 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: expr 10 - 1 2024-09-16T05:24:07.924 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: seq 0 9 2024-09-16T05:24:07.936 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: for count in $(seq 0 $(expr $retries - 1)) 2024-09-16T05:24:07.936 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:340: TEST_profile_disallow_builtin_params_modify: errors=0 2024-09-16T05:24:07.936 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:341: TEST_profile_disallow_builtin_params_modify: sleep 2 2024-09-16T05:24:09.937 INFO:tasks.workunit.client.0.smithi178.stdout:Check configuration values - Attempt#: 0 2024-09-16T05:24:09.938 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:343: TEST_profile_disallow_builtin_params_modify: echo 'Check configuration values - Attempt#: 0' 2024-09-16T05:24:09.938 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_scheduler_background_recovery_res 2024-09-16T05:24:10.321 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: local res=0.000000 2024-09-16T05:24:10.322 INFO:tasks.workunit.client.0.smithi178.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000 2024-09-16T05:24:10.322 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:346: TEST_profile_disallow_builtin_params_modify: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000' 2024-09-16T05:24:10.322 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: echo '0.000000 == .800000' 2024-09-16T05:24:10.322 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-16T05:24:10.323 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: (( 0 )) 2024-09-16T05:24:10.324 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: ceph config show osd.0 2024-09-16T05:24:10.324 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: grep osd_mclock_scheduler_background_recovery_res 2024-09-16T05:24:10.324 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: awk '{ print $2 }' 2024-09-16T05:24:10.324 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: bc 2024-09-16T05:24:10.691 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: res=.700000 2024-09-16T05:24:10.691 INFO:tasks.workunit.client.0.smithi178.stdout:Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000 2024-09-16T05:24:10.692 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:354: TEST_profile_disallow_builtin_params_modify: echo 'Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000' 2024-09-16T05:24:10.692 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.700000 == .800000' 2024-09-16T05:24:10.693 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-16T05:24:10.694 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.700000 != 0.700000' 2024-09-16T05:24:10.694 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-16T05:24:10.694 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-09-16T05:24:10.697 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_background_recovery_res 2024-09-16T05:24:10.697 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: bc 2024-09-16T05:24:10.697 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-09-16T05:24:10.697 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:24:10.697 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:24:10.698 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:24:10.698 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:24:10.698 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:24:10.698 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-osd.0.asok 2024-09-16T05:24:10.698 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-09-16T05:24:10.698 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.26942/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-09-16T05:24:10.779 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: res=0.700000 2024-09-16T05:24:10.780 INFO:tasks.workunit.client.0.smithi178.stdout:Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000 2024-09-16T05:24:10.781 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:363: TEST_profile_disallow_builtin_params_modify: echo 'Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000' 2024-09-16T05:24:10.781 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.700000 == .800000' 2024-09-16T05:24:10.781 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-16T05:24:10.782 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.700000 != 0.700000' 2024-09-16T05:24:10.782 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-16T05:24:10.783 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-09-16T05:24:10.783 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:370: TEST_profile_disallow_builtin_params_modify: '[' 0 -eq 0 ']' 2024-09-16T05:24:10.783 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:372: TEST_profile_disallow_builtin_params_modify: break 2024-09-16T05:24:10.783 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:329: TEST_profile_disallow_builtin_params_modify: for opt in "${options[@]}" 2024-09-16T05:24:10.785 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_client_res 2024-09-16T05:24:10.785 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: bc 2024-09-16T05:24:10.785 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-09-16T05:24:10.785 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:24:10.785 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:24:10.785 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:24:10.785 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:24:10.785 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:24:10.785 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-osd.0.asok 2024-09-16T05:24:10.786 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-09-16T05:24:10.786 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.26942/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-16T05:24:10.867 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: local opt_val_orig=0.300000 2024-09-16T05:24:10.867 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: echo '0.300000 + 0.1' 2024-09-16T05:24:10.867 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-16T05:24:10.868 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: local opt_val_new=.400000 2024-09-16T05:24:10.869 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:335: TEST_profile_disallow_builtin_params_modify: ceph config set osd.0 osd_mclock_scheduler_client_res .400000 2024-09-16T05:24:11.427 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: expr 10 - 1 2024-09-16T05:24:11.428 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: seq 0 9 2024-09-16T05:24:11.429 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: for count in $(seq 0 $(expr $retries - 1)) 2024-09-16T05:24:11.429 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:340: TEST_profile_disallow_builtin_params_modify: errors=0 2024-09-16T05:24:11.429 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:341: TEST_profile_disallow_builtin_params_modify: sleep 2 2024-09-16T05:24:13.430 INFO:tasks.workunit.client.0.smithi178.stdout:Check configuration values - Attempt#: 0 2024-09-16T05:24:13.431 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:343: TEST_profile_disallow_builtin_params_modify: echo 'Check configuration values - Attempt#: 0' 2024-09-16T05:24:13.431 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-09-16T05:24:13.860 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: local res=0.000000 2024-09-16T05:24:13.861 INFO:tasks.workunit.client.0.smithi178.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000 2024-09-16T05:24:13.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:346: TEST_profile_disallow_builtin_params_modify: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000' 2024-09-16T05:24:13.861 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: echo '0.000000 == .400000' 2024-09-16T05:24:13.861 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-16T05:24:13.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: (( 0 )) 2024-09-16T05:24:13.862 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: ceph config show osd.0 2024-09-16T05:24:13.862 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: grep osd_mclock_scheduler_client_res 2024-09-16T05:24:13.863 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: awk '{ print $2 }' 2024-09-16T05:24:13.863 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: bc 2024-09-16T05:24:14.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: res=.300000 2024-09-16T05:24:14.231 INFO:tasks.workunit.client.0.smithi178.stdout:Running config: osd.0 osd_mclock_scheduler_client_res = .300000 2024-09-16T05:24:14.232 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:354: TEST_profile_disallow_builtin_params_modify: echo 'Running config: osd.0 osd_mclock_scheduler_client_res = .300000' 2024-09-16T05:24:14.232 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.300000 == .400000' 2024-09-16T05:24:14.232 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-16T05:24:14.232 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.300000 != 0.300000' 2024-09-16T05:24:14.232 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-16T05:24:14.233 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-09-16T05:24:14.234 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_client_res 2024-09-16T05:24:14.234 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: bc 2024-09-16T05:24:14.235 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-09-16T05:24:14.235 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:24:14.235 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:24:14.235 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:24:14.235 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:24:14.235 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:24:14.235 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-osd.0.asok 2024-09-16T05:24:14.235 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-09-16T05:24:14.236 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.26942/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-16T05:24:14.317 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: res=0.300000 2024-09-16T05:24:14.318 INFO:tasks.workunit.client.0.smithi178.stdout:Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000 2024-09-16T05:24:14.318 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:363: TEST_profile_disallow_builtin_params_modify: echo 'Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000' 2024-09-16T05:24:14.318 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.300000 == .400000' 2024-09-16T05:24:14.318 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-16T05:24:14.319 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.300000 != 0.300000' 2024-09-16T05:24:14.320 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-16T05:24:14.321 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-09-16T05:24:14.322 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:370: TEST_profile_disallow_builtin_params_modify: '[' 0 -eq 0 ']' 2024-09-16T05:24:14.322 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:372: TEST_profile_disallow_builtin_params_modify: break 2024-09-16T05:24:14.322 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:380: TEST_profile_disallow_builtin_params_modify: teardown td/mclock-config 2024-09-16T05:24:14.322 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T05:24:14.322 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:24:14.322 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T05:24:14.322 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:24:14.322 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:24:14.323 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:24:14.323 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:24:14.323 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:24:14.427 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:24:14.427 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:24:14.428 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:24:14.428 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:24:14.429 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:24:14.429 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:24:14.430 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:24:14.431 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:24:14.431 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:24:14.431 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:24:14.432 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:24:14.432 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:24:14.433 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:24:14.433 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T05:24:14.438 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:24:14.438 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:24:14.438 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:24:14.439 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26942 2024-09-16T05:24:14.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:24:14.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:24:14.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-09-16T05:24:14.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T05:24:14.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:24:14.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T05:24:14.441 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:24:14.441 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:24:14.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:24:14.442 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:24:14.442 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:24:14.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:24:14.443 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:24:14.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:24:14.443 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:24:14.444 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:24:14.445 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:24:14.445 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:24:14.445 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:24:14.445 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:24:14.446 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:24:14.446 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:24:14.446 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:24:14.446 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:24:14.447 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T05:24:14.448 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:24:14.448 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:24:14.448 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:24:14.448 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26942 2024-09-16T05:24:14.450 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:24:14.450 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:24:14.450 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-09-16T05:24:14.450 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-09-16T05:24:14.450 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-16T05:24:14.450 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-16T05:24:14.451 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T05:24:14.451 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:24:14.451 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T05:24:14.451 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:24:14.451 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:24:14.451 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:24:14.451 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:24:14.451 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:24:14.452 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:24:14.453 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:24:14.453 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:24:14.453 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:24:14.453 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:24:14.453 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:24:14.453 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:24:14.454 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:24:14.454 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:24:14.454 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:24:14.454 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:24:14.455 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:24:14.456 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:24:14.456 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T05:24:14.457 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:24:14.457 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:24:14.457 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:24:14.457 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26942 2024-09-16T05:24:14.458 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:24:14.458 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:24:14.459 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-16T05:24:14.459 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:24:14.460 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:24:14.460 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:24:14.460 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26942 2024-09-16T05:24:14.462 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:24:14.462 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:24:14.462 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:24:14.462 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-16T05:24:14.462 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_disallow_builtin_params_override td/mclock-config 2024-09-16T05:24:14.462 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:384: TEST_profile_disallow_builtin_params_override: local dir=td/mclock-config 2024-09-16T05:24:14.462 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:386: TEST_profile_disallow_builtin_params_override: setup td/mclock-config 2024-09-16T05:24:14.462 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-16T05:24:14.463 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-16T05:24:14.463 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T05:24:14.463 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:24:14.463 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T05:24:14.463 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:24:14.463 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:24:14.463 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:24:14.463 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:24:14.463 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:24:14.464 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:24:14.464 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:24:14.464 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:24:14.464 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:24:14.465 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:24:14.465 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:24:14.466 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:24:14.466 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:24:14.466 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:24:14.467 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:24:14.467 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:24:14.468 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:24:14.469 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:24:14.469 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T05:24:14.470 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:24:14.470 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:24:14.470 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:24:14.470 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26942 2024-09-16T05:24:14.471 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:24:14.471 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:24:14.471 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-16T05:24:14.472 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:24:14.472 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:24:14.472 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:24:14.473 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26942 2024-09-16T05:24:14.474 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:24:14.474 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:24:14.474 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:24:14.475 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-16T05:24:14.475 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:387: TEST_profile_disallow_builtin_params_override: run_mon td/mclock-config a 2024-09-16T05:24:14.475 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-09-16T05:24:14.475 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:24:14.475 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:24:14.475 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:24:14.475 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-09-16T05:24:14.475 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-09-16T05:24:14.849 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:24:14.849 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:24:14.849 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:24:14.849 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:24:14.850 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:24:14.850 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:24:14.850 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26942/$cluster-$name.asok' 2024-09-16T05:24:14.850 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26942/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T05:24:15.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:24:15.043 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:24:15.043 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:24:15.043 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:24:15.043 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:24:15.043 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:24:15.043 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:24:15.043 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:24:15.043 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:24:15.044 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:24:15.044 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:24:15.045 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:24:15.045 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-mon.a.asok 2024-09-16T05:24:15.045 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:24:15.045 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26942/ceph-mon.a.asok config get fsid 2024-09-16T05:24:15.123 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:24:15.124 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:24:15.124 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:24:15.124 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:24:15.124 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:24:15.124 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:24:15.124 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:24:15.124 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:24:15.124 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:24:15.124 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:24:15.124 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:24:15.125 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-mon.a.asok 2024-09-16T05:24:15.125 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:24:15.125 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26942/ceph-mon.a.asok config get mon_host 2024-09-16T05:24:15.201 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:388: TEST_profile_disallow_builtin_params_override: run_mgr td/mclock-config x 2024-09-16T05:24:15.201 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-09-16T05:24:15.201 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T05:24:15.202 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T05:24:15.202 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T05:24:15.202 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-09-16T05:24:15.202 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T05:24:16.045 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T05:24:16.045 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:24:16.045 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:24:16.045 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:24:16.046 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:24:16.046 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:24:16.046 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26942/$cluster-$name.asok' 2024-09-16T05:24:16.046 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:24:16.046 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26942/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:24:16.076 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:390: TEST_profile_disallow_builtin_params_override: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-09-16T05:24:16.076 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-09-16T05:24:16.076 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:24:16.076 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T05:24:16.076 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:24:16.076 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-09-16T05:24:16.076 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=64e3af7b-5a84-4040-b122-b03cebe716a6 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-16T05:24:16.076 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:24:16.076 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:24:16.076 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:24:16.077 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-16T05:24:16.077 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-16T05:24:16.077 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:24:16.077 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:24:16.077 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-16T05:24:16.077 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:24:16.077 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:24:16.077 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:24:16.077 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:24:16.078 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:24:16.078 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:24:16.078 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26942/$cluster-$name.asok' 2024-09-16T05:24:16.078 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26942/$cluster-$name.asok' 2024-09-16T05:24:16.078 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:24:16.078 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:24:16.078 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:24:16.078 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-16T05:24:16.078 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-16T05:24:16.079 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:24:16.079 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:24:16.079 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:24:16.079 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:24:16.079 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:24:16.079 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-16T05:24:16.079 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-09-16T05:24:16.079 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:24:16.079 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dbfec7cc-ff03-4265-9098-231931b552df 2024-09-16T05:24:16.080 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 dbfec7cc-ff03-4265-9098-231931b552df 2024-09-16T05:24:16.080 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 dbfec7cc-ff03-4265-9098-231931b552df' 2024-09-16T05:24:16.080 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:24:16.093 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAAwedmJuiDBRAA8fl92tbpp3avchfkv5QYqA== 2024-09-16T05:24:16.093 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAAwedmJuiDBRAA8fl92tbpp3avchfkv5QYqA=="}' 2024-09-16T05:24:16.093 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dbfec7cc-ff03-4265-9098-231931b552df -i td/mclock-config/0/new.json 2024-09-16T05:24:16.390 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:24:16.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-09-16T05:24:16.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=64e3af7b-5a84-4040-b122-b03cebe716a6 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26942/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQAAwedmJuiDBRAA8fl92tbpp3avchfkv5QYqA== --osd-uuid dbfec7cc-ff03-4265-9098-231931b552df 2024-09-16T05:24:16.429 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:24:16.428+0000 7f129b211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:24:16.459 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:24:16.458+0000 7f129b211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:24:16.460 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:24:16.459+0000 7f129b211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:24:16.460 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:24:16.460+0000 7f129b211640 -1 bdev(0x560a52267c00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2024-09-16T05:24:16.461 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:24:16.460+0000 7f129b211640 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-09-16T05:24:22.135 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-09-16T05:24:22.136 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:24:22.137 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T05:24:22.137 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T05:24:22.137 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:24:22.646 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:24:22.647 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T05:24:22.647 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=64e3af7b-5a84-4040-b122-b03cebe716a6 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26942/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-16T05:24:22.647 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:24:22.647 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:24:22.647 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:24:22.673 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:24:22.672+0000 7efc5ec0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:24:22.675 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:24:22.674+0000 7efc5ec0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:24:22.676 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:24:22.675+0000 7efc5ec0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:24:23.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T05:24:23.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:24:23.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:24:23.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:24:23.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:24:23.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:24:23.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:24:23.043 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:24:23.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:24:23.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:24:23.415 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:24:23.761 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:24:23.760+0000 7efc5ec0b640 -1 Falling back to public interface 2024-09-16T05:24:24.416 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:24:24.416 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:24:24.417 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:24:24.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:24:24.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:24:24.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:24:24.817 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:24:25.818 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:24:25.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:24:25.819 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:24:25.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:24:25.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:24:25.820 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:24:26.235 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:24:26.888 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:24:26.887+0000 7efc5ec0b640 -1 osd.0 0 log_to_monitors true 2024-09-16T05:24:27.236 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:24:27.236 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:24:27.236 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:24:27.237 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:24:27.237 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:24:27.237 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:24:27.614 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:24:28.615 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:24:28.615 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:24:28.616 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:24:28.616 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:24:28.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:24:28.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:24:29.028 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:24:30.028 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:24:30.029 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:24:30.029 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:24:30.029 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:24:30.029 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:24:30.029 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:24:30.435 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:24:31.437 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:24:31.437 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:24:31.437 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:24:31.438 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:24:31.438 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:24:31.438 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:24:31.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:24:32.934 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:24:32.934 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:24:32.935 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:24:32.935 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:24:32.935 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:24:32.935 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:24:33.338 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:24:34.340 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:24:34.340 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:24:34.340 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:24:34.341 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:24:34.341 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:24:34.341 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:24:34.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:24:35.791 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:24:35.791 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:24:35.791 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:24:35.791 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:24:35.792 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:24:35.792 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:24:36.215 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:24:37.216 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:24:37.216 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:24:37.217 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:24:37.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:24:37.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:24:37.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:24:37.626 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:24:38.628 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:24:38.628 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:24:38.628 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:24:38.629 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:24:38.629 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:24:38.629 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:24:39.313 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:24:40.314 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:24:40.314 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:24:40.314 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:24:40.315 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:24:40.315 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:24:40.315 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:24:41.135 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:24:42.136 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:24:42.136 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:24:42.137 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:24:42.137 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:24:42.137 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:24:42.137 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:24:42.521 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:24:43.522 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:24:43.523 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:24:43.523 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:24:43.523 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:24:43.524 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:24:43.524 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:24:44.257 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:24:45.259 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:24:45.259 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:24:45.259 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:24:45.260 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:24:45.260 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:24:45.260 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:24:45.893 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:24:46.895 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:24:46.895 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:24:46.895 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:24:46.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:24:46.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:24:46.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:24:48.051 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:24:49.052 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:24:49.052 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:24:49.052 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:24:49.053 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:24:49.053 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:24:49.053 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:24:49.519 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/168411236,v1:127.0.0.1:6803/168411236] [v2:127.0.0.1:6804/168411236,v1:127.0.0.1:6805/168411236] exists,up dbfec7cc-ff03-4265-9098-231931b552df 2024-09-16T05:24:49.520 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:24:49.520 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:24:49.520 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:24:49.520 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:393: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_profile 2024-09-16T05:24:49.895 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:393: TEST_profile_disallow_builtin_params_override: local def_mclock_profile=balanced 2024-09-16T05:24:49.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:394: TEST_profile_disallow_builtin_params_override: test balanced = balanced 2024-09-16T05:24:49.897 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_profile 2024-09-16T05:24:49.897 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-09-16T05:24:49.897 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:24:49.897 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:24:49.897 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:24:49.897 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:24:49.897 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:24:49.897 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-osd.0.asok 2024-09-16T05:24:49.897 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-09-16T05:24:49.898 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.26942/ceph-osd.0.asok config get osd_mclock_profile 2024-09-16T05:24:49.982 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: local 'cur_mclock_profile="high_recovery_ops"' 2024-09-16T05:24:49.983 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: eval echo '"high_recovery_ops"' 2024-09-16T05:24:49.983 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: echo high_recovery_ops 2024-09-16T05:24:49.983 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: cur_mclock_profile=high_recovery_ops 2024-09-16T05:24:49.983 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:401: TEST_profile_disallow_builtin_params_override: test high_recovery_ops = high_recovery_ops 2024-09-16T05:24:49.983 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:403: TEST_profile_disallow_builtin_params_override: options=('osd_mclock_scheduler_background_recovery_res' 'osd_mclock_scheduler_client_res') 2024-09-16T05:24:49.983 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:403: TEST_profile_disallow_builtin_params_override: declare -a options 2024-09-16T05:24:49.983 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:406: TEST_profile_disallow_builtin_params_override: local retries=10 2024-09-16T05:24:49.983 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:407: TEST_profile_disallow_builtin_params_override: local errors=0 2024-09-16T05:24:49.983 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:408: TEST_profile_disallow_builtin_params_override: for opt in "${options[@]}" 2024-09-16T05:24:49.983 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_background_recovery_res 2024-09-16T05:24:49.983 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-09-16T05:24:49.983 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: bc 2024-09-16T05:24:49.983 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:24:49.984 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:24:49.984 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:24:49.984 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:24:49.984 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:24:49.984 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-osd.0.asok 2024-09-16T05:24:49.984 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-09-16T05:24:49.984 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.26942/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-09-16T05:24:50.066 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: local opt_val_orig=0.700000 2024-09-16T05:24:50.067 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: echo '0.700000 + 0.1' 2024-09-16T05:24:50.067 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-16T05:24:50.068 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: local opt_val_new=.800000 2024-09-16T05:24:50.068 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:414: TEST_profile_disallow_builtin_params_override: ceph tell osd.0 config set osd_mclock_scheduler_background_recovery_res .800000 2024-09-16T05:24:50.170 INFO:tasks.workunit.client.0.smithi178.stdout:{ 2024-09-16T05:24:50.170 INFO:tasks.workunit.client.0.smithi178.stdout: "success": "osd_mclock_scheduler_background_recovery_res = '' " 2024-09-16T05:24:50.170 INFO:tasks.workunit.client.0.smithi178.stdout:} 2024-09-16T05:24:50.179 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: expr 10 - 1 2024-09-16T05:24:50.180 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: seq 0 9 2024-09-16T05:24:50.181 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: for count in $(seq 0 $(expr $retries - 1)) 2024-09-16T05:24:50.181 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:419: TEST_profile_disallow_builtin_params_override: errors=0 2024-09-16T05:24:50.181 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:420: TEST_profile_disallow_builtin_params_override: sleep 2 2024-09-16T05:24:52.182 INFO:tasks.workunit.client.0.smithi178.stdout:Check configuration values - Attempt#: 0 2024-09-16T05:24:52.183 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:422: TEST_profile_disallow_builtin_params_override: echo 'Check configuration values - Attempt#: 0' 2024-09-16T05:24:52.183 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_scheduler_background_recovery_res 2024-09-16T05:24:52.558 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: local res=0.000000 2024-09-16T05:24:52.558 INFO:tasks.workunit.client.0.smithi178.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000 2024-09-16T05:24:52.558 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:425: TEST_profile_disallow_builtin_params_override: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000' 2024-09-16T05:24:52.559 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: echo '0.000000 == .800000' 2024-09-16T05:24:52.559 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-16T05:24:52.559 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: (( 0 )) 2024-09-16T05:24:52.560 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: ceph config show osd.0 2024-09-16T05:24:52.560 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: grep osd_mclock_scheduler_background_recovery_res 2024-09-16T05:24:52.560 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: awk '{ print $2 }' 2024-09-16T05:24:52.560 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: bc 2024-09-16T05:24:52.930 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: res=.700000 2024-09-16T05:24:52.930 INFO:tasks.workunit.client.0.smithi178.stdout:Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000 2024-09-16T05:24:52.931 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:433: TEST_profile_disallow_builtin_params_override: echo 'Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000' 2024-09-16T05:24:52.931 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.700000 == .800000' 2024-09-16T05:24:52.931 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-16T05:24:52.932 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.700000 != 0.700000' 2024-09-16T05:24:52.932 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-16T05:24:52.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-09-16T05:24:52.934 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_background_recovery_res 2024-09-16T05:24:52.934 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-09-16T05:24:52.935 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:24:52.935 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: bc 2024-09-16T05:24:52.935 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:24:52.935 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:24:52.935 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:24:52.935 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:24:52.935 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-osd.0.asok 2024-09-16T05:24:52.935 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-09-16T05:24:52.935 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.26942/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-09-16T05:24:53.018 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: res=0.700000 2024-09-16T05:24:53.018 INFO:tasks.workunit.client.0.smithi178.stdout:Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000 2024-09-16T05:24:53.019 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:442: TEST_profile_disallow_builtin_params_override: echo 'Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000' 2024-09-16T05:24:53.019 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.700000 == .800000' 2024-09-16T05:24:53.019 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-16T05:24:53.020 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.700000 != 0.700000' 2024-09-16T05:24:53.020 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-16T05:24:53.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-09-16T05:24:53.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:449: TEST_profile_disallow_builtin_params_override: '[' 0 -eq 0 ']' 2024-09-16T05:24:53.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:451: TEST_profile_disallow_builtin_params_override: break 2024-09-16T05:24:53.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:408: TEST_profile_disallow_builtin_params_override: for opt in "${options[@]}" 2024-09-16T05:24:53.023 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_client_res 2024-09-16T05:24:53.023 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-09-16T05:24:53.023 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: bc 2024-09-16T05:24:53.023 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:24:53.023 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:24:53.023 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:24:53.023 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:24:53.023 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:24:53.023 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-osd.0.asok 2024-09-16T05:24:53.023 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-09-16T05:24:53.023 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.26942/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-16T05:24:53.105 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: local opt_val_orig=0.300000 2024-09-16T05:24:53.106 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: echo '0.300000 + 0.1' 2024-09-16T05:24:53.106 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-16T05:24:53.107 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: local opt_val_new=.400000 2024-09-16T05:24:53.107 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:414: TEST_profile_disallow_builtin_params_override: ceph tell osd.0 config set osd_mclock_scheduler_client_res .400000 2024-09-16T05:24:53.248 INFO:tasks.workunit.client.0.smithi178.stdout:{ 2024-09-16T05:24:53.248 INFO:tasks.workunit.client.0.smithi178.stdout: "success": "osd_mclock_scheduler_client_res = '' " 2024-09-16T05:24:53.248 INFO:tasks.workunit.client.0.smithi178.stdout:} 2024-09-16T05:24:53.257 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: expr 10 - 1 2024-09-16T05:24:53.258 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: seq 0 9 2024-09-16T05:24:53.259 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: for count in $(seq 0 $(expr $retries - 1)) 2024-09-16T05:24:53.259 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:419: TEST_profile_disallow_builtin_params_override: errors=0 2024-09-16T05:24:53.259 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:420: TEST_profile_disallow_builtin_params_override: sleep 2 2024-09-16T05:24:55.261 INFO:tasks.workunit.client.0.smithi178.stdout:Check configuration values - Attempt#: 0 2024-09-16T05:24:55.261 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:422: TEST_profile_disallow_builtin_params_override: echo 'Check configuration values - Attempt#: 0' 2024-09-16T05:24:55.261 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-09-16T05:24:55.646 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: local res=0.000000 2024-09-16T05:24:55.646 INFO:tasks.workunit.client.0.smithi178.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000 2024-09-16T05:24:55.646 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:425: TEST_profile_disallow_builtin_params_override: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000' 2024-09-16T05:24:55.647 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: echo '0.000000 == .400000' 2024-09-16T05:24:55.647 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-16T05:24:55.647 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: (( 0 )) 2024-09-16T05:24:55.648 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: ceph config show osd.0 2024-09-16T05:24:55.648 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: grep osd_mclock_scheduler_client_res 2024-09-16T05:24:55.648 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: awk '{ print $2 }' 2024-09-16T05:24:55.648 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: bc 2024-09-16T05:24:56.016 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: res=.300000 2024-09-16T05:24:56.016 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:433: TEST_profile_disallow_builtin_params_override: echo 'Running config: osd.0 osd_mclock_scheduler_client_res = .300000' 2024-09-16T05:24:56.016 INFO:tasks.workunit.client.0.smithi178.stdout:Running config: osd.0 osd_mclock_scheduler_client_res = .300000 2024-09-16T05:24:56.017 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.300000 == .400000' 2024-09-16T05:24:56.017 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-16T05:24:56.018 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.300000 != 0.300000' 2024-09-16T05:24:56.018 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-16T05:24:56.019 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-09-16T05:24:56.020 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_client_res 2024-09-16T05:24:56.020 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: bc 2024-09-16T05:24:56.020 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-09-16T05:24:56.020 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:24:56.021 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:24:56.021 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:24:56.021 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:24:56.021 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:24:56.021 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-osd.0.asok 2024-09-16T05:24:56.021 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-09-16T05:24:56.021 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.26942/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-16T05:24:56.103 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: res=0.300000 2024-09-16T05:24:56.104 INFO:tasks.workunit.client.0.smithi178.stdout:Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000 2024-09-16T05:24:56.104 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:442: TEST_profile_disallow_builtin_params_override: echo 'Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000' 2024-09-16T05:24:56.104 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.300000 == .400000' 2024-09-16T05:24:56.104 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-16T05:24:56.105 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.300000 != 0.300000' 2024-09-16T05:24:56.106 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-16T05:24:56.107 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-09-16T05:24:56.108 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:449: TEST_profile_disallow_builtin_params_override: '[' 0 -eq 0 ']' 2024-09-16T05:24:56.108 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:451: TEST_profile_disallow_builtin_params_override: break 2024-09-16T05:24:56.108 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:459: TEST_profile_disallow_builtin_params_override: teardown td/mclock-config 2024-09-16T05:24:56.108 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T05:24:56.108 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:24:56.108 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T05:24:56.108 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:24:56.108 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:24:56.108 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:24:56.108 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:24:56.108 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:24:56.213 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:24:56.213 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:24:56.214 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:24:56.214 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:24:56.215 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:24:56.215 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:24:56.216 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:24:56.216 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:24:56.216 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:24:56.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:24:56.217 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:24:56.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:24:56.218 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:24:56.218 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T05:24:56.224 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:24:56.224 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:24:56.224 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:24:56.224 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26942 2024-09-16T05:24:56.225 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:24:56.226 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:24:56.226 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-09-16T05:24:56.226 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T05:24:56.226 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:24:56.226 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T05:24:56.226 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:24:56.226 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:24:56.226 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:24:56.226 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:24:56.226 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:24:56.227 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:24:56.227 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:24:56.228 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:24:56.228 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:24:56.229 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:24:56.229 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:24:56.229 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:24:56.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:24:56.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:24:56.231 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:24:56.231 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:24:56.231 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:24:56.232 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:24:56.233 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T05:24:56.233 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:24:56.233 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:24:56.233 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:24:56.234 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26942 2024-09-16T05:24:56.235 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:24:56.235 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:24:56.235 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-09-16T05:24:56.235 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-09-16T05:24:56.236 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-16T05:24:56.236 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-16T05:24:56.236 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T05:24:56.236 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:24:56.236 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T05:24:56.236 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:24:56.236 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:24:56.236 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:24:56.236 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:24:56.236 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:24:56.237 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:24:56.237 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:24:56.237 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:24:56.238 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:24:56.239 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:24:56.239 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:24:56.239 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:24:56.240 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:24:56.240 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:24:56.240 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:24:56.240 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:24:56.241 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:24:56.241 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:24:56.242 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T05:24:56.243 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:24:56.243 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:24:56.243 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:24:56.243 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26942 2024-09-16T05:24:56.244 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:24:56.244 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:24:56.244 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-16T05:24:56.245 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:24:56.245 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:24:56.245 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:24:56.245 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26942 2024-09-16T05:24:56.247 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:24:56.247 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:24:56.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:24:56.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-16T05:24:56.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_recovery_limit_adjustment_mclock td/mclock-config 2024-09-16T05:24:56.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:202: TEST_recovery_limit_adjustment_mclock: local dir=td/mclock-config 2024-09-16T05:24:56.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:204: TEST_recovery_limit_adjustment_mclock: setup td/mclock-config 2024-09-16T05:24:56.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-16T05:24:56.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-16T05:24:56.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T05:24:56.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:24:56.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T05:24:56.248 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:24:56.248 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:24:56.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:24:56.249 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:24:56.249 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:24:56.249 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:24:56.249 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:24:56.250 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:24:56.250 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:24:56.251 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:24:56.251 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:24:56.251 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:24:56.252 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:24:56.252 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:24:56.252 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:24:56.252 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:24:56.253 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:24:56.254 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:24:56.254 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T05:24:56.255 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:24:56.255 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:24:56.255 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:24:56.255 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26942 2024-09-16T05:24:56.256 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:24:56.256 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:24:56.256 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-16T05:24:56.257 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:24:56.257 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:24:56.257 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:24:56.257 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26942 2024-09-16T05:24:56.259 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:24:56.259 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:24:56.259 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:24:56.259 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-16T05:24:56.259 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:205: TEST_recovery_limit_adjustment_mclock: run_mon td/mclock-config a 2024-09-16T05:24:56.260 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-09-16T05:24:56.260 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:24:56.260 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:24:56.260 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:24:56.260 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-09-16T05:24:56.260 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-09-16T05:24:56.623 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:24:56.623 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:24:56.623 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:24:56.623 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:24:56.623 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:24:56.623 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:24:56.623 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26942/$cluster-$name.asok' 2024-09-16T05:24:56.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26942/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T05:24:56.815 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:24:56.816 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:24:56.816 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:24:56.816 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:24:56.816 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:24:56.816 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:24:56.816 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:24:56.816 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:24:56.816 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:24:56.817 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:24:56.817 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:24:56.817 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:24:56.818 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-mon.a.asok 2024-09-16T05:24:56.818 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:24:56.818 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26942/ceph-mon.a.asok config get fsid 2024-09-16T05:24:56.897 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:24:56.897 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:24:56.897 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:24:56.897 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:24:56.897 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:24:56.897 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:24:56.897 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:24:56.897 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:24:56.897 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:24:56.897 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:24:56.898 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:24:56.898 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-mon.a.asok 2024-09-16T05:24:56.898 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:24:56.898 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26942/ceph-mon.a.asok config get mon_host 2024-09-16T05:24:56.975 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:206: TEST_recovery_limit_adjustment_mclock: run_mgr td/mclock-config x 2024-09-16T05:24:56.975 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-09-16T05:24:56.975 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T05:24:56.975 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T05:24:56.975 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T05:24:56.975 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-09-16T05:24:56.975 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T05:24:57.810 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T05:24:57.810 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:24:57.810 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:24:57.810 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:24:57.810 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:24:57.810 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:24:57.810 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26942/$cluster-$name.asok' 2024-09-16T05:24:57.811 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:24:57.811 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26942/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:24:57.840 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:208: TEST_recovery_limit_adjustment_mclock: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-09-16T05:24:57.840 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-09-16T05:24:57.840 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:24:57.840 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T05:24:57.840 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:24:57.840 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-09-16T05:24:57.841 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=64e3af7b-5a84-4040-b122-b03cebe716a6 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-16T05:24:57.841 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:24:57.841 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:24:57.841 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:24:57.841 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-16T05:24:57.841 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-16T05:24:57.841 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:24:57.841 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:24:57.841 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-16T05:24:57.841 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:24:57.841 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:24:57.841 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:24:57.841 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:24:57.842 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:24:57.842 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:24:57.842 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26942/$cluster-$name.asok' 2024-09-16T05:24:57.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26942/$cluster-$name.asok' 2024-09-16T05:24:57.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:24:57.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:24:57.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:24:57.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-16T05:24:57.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-16T05:24:57.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:24:57.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:24:57.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:24:57.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:24:57.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:24:57.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-16T05:24:57.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-09-16T05:24:57.843 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:24:57.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0cd3be0f-7747-44a5-8697-90126c407bae 2024-09-16T05:24:57.844 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 0cd3be0f-7747-44a5-8697-90126c407bae 2024-09-16T05:24:57.844 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0cd3be0f-7747-44a5-8697-90126c407bae' 2024-09-16T05:24:57.844 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:24:57.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQApwedmJcAYMxAAQxzDH4WL+TJv0KU3Jd/FpA== 2024-09-16T05:24:57.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQApwedmJcAYMxAAQxzDH4WL+TJv0KU3Jd/FpA=="}' 2024-09-16T05:24:57.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0cd3be0f-7747-44a5-8697-90126c407bae -i td/mclock-config/0/new.json 2024-09-16T05:24:58.164 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:24:58.173 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-09-16T05:24:58.174 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=64e3af7b-5a84-4040-b122-b03cebe716a6 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26942/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQApwedmJcAYMxAAQxzDH4WL+TJv0KU3Jd/FpA== --osd-uuid 0cd3be0f-7747-44a5-8697-90126c407bae 2024-09-16T05:24:58.202 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:24:58.200+0000 7f52bad89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:24:58.224 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:24:58.222+0000 7f52bad89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:24:58.225 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:24:58.223+0000 7f52bad89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:24:58.226 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:24:58.223+0000 7f52bad89640 -1 bdev(0x555d87779c00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2024-09-16T05:24:58.226 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:24:58.224+0000 7f52bad89640 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-09-16T05:25:03.593 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-09-16T05:25:03.593 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:25:03.593 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T05:25:03.594 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T05:25:03.594 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:25:04.113 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:25:04.114 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T05:25:04.114 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=64e3af7b-5a84-4040-b122-b03cebe716a6 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26942/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-16T05:25:04.114 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:25:04.114 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:25:04.115 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:25:04.140 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:25:04.139+0000 7f484d7bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:25:04.142 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:25:04.141+0000 7f484d7bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:25:04.143 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:25:04.142+0000 7f484d7bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:25:04.487 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T05:25:04.487 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:25:04.487 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:25:04.487 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:25:04.487 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:25:04.487 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:25:04.488 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:25:04.488 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:25:04.488 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:25:04.488 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:25:04.890 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:25:05.734 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:25:05.733+0000 7f484d7bb640 -1 Falling back to public interface 2024-09-16T05:25:05.891 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:25:05.892 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:25:05.892 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:25:05.892 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:25:05.892 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:25:05.892 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:25:06.308 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:25:07.309 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:25:07.309 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:25:07.309 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:25:07.310 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:25:07.310 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:25:07.310 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:25:07.725 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:25:08.370 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:25:08.369+0000 7f484d7bb640 -1 osd.0 0 log_to_monitors true 2024-09-16T05:25:08.726 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:25:08.727 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:25:08.727 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:25:08.727 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:25:08.727 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:25:08.728 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:25:09.097 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:25:10.099 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:25:10.099 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:25:10.099 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:25:10.099 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:25:10.100 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:25:10.100 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:25:10.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:25:11.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:25:11.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:25:11.486 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:25:11.486 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:25:11.486 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:25:11.486 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:25:11.878 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:25:12.879 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:25:12.879 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:25:12.879 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:25:12.880 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:25:12.880 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:25:12.880 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:25:13.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:25:14.396 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:25:14.396 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:25:14.397 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:25:14.397 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:25:14.397 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:25:14.397 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:25:14.792 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:25:15.793 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:25:15.793 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:25:15.794 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:25:15.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:25:15.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:25:15.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:25:16.236 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:25:17.237 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:25:17.237 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:25:17.238 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:25:17.238 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:25:17.238 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:25:17.238 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:25:17.650 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:25:18.652 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:25:18.652 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:25:18.652 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:25:18.652 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:25:18.653 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:25:18.653 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:25:19.083 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:25:20.085 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:25:20.085 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:25:20.085 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:25:20.086 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:25:20.086 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:25:20.086 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:25:20.927 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:25:21.929 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:25:21.929 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:25:21.929 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:25:21.930 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:25:21.930 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:25:21.930 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:25:22.792 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:25:23.793 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:25:23.793 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:25:23.794 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:25:23.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:25:23.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:25:23.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:25:24.287 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:25:25.288 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:25:25.288 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:25:25.288 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:25:25.289 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:25:25.289 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:25:25.289 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:25:26.066 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:25:27.067 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:25:27.067 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:25:27.068 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:25:27.068 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:25:27.068 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:25:27.068 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:25:27.786 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:25:28.787 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:25:28.787 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:25:28.788 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:25:28.788 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:25:28.788 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:25:28.788 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:25:29.663 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:25:30.664 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:25:30.664 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:25:30.664 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:25:30.665 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:25:30.665 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:25:30.665 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:25:31.073 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/722297907,v1:127.0.0.1:6803/722297907] [v2:127.0.0.1:6804/722297907,v1:127.0.0.1:6805/722297907] exists,up 0cd3be0f-7747-44a5-8697-90126c407bae 2024-09-16T05:25:31.075 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:25:31.075 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:25:31.075 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:25:31.075 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2024-09-16T05:25:31.075 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:25:31.075 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:25:31.076 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:25:31.076 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:25:31.076 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:25:31.076 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-osd.0.asok 2024-09-16T05:25:31.076 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2024-09-16T05:25:31.076 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26942/ceph-osd.0.asok config get osd_recovery_max_active 2024-09-16T05:25:31.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: local 'recoveries={"osd_recovery_max_active":"0"}' 2024-09-16T05:25:31.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:212: TEST_recovery_limit_adjustment_mclock: echo '{"osd_recovery_max_active":"0"}' 2024-09-16T05:25:31.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:212: TEST_recovery_limit_adjustment_mclock: grep --quiet osd_recovery_max_active 2024-09-16T05:25:31.160 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:217: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_recovery_max_active 10 2024-09-16T05:25:31.753 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:218: TEST_recovery_limit_adjustment_mclock: sleep 2 2024-09-16T05:25:33.756 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2024-09-16T05:25:33.756 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:25:33.756 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:25:33.756 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:25:33.757 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:25:33.757 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:25:33.757 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-osd.0.asok 2024-09-16T05:25:33.757 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2024-09-16T05:25:33.757 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26942/ceph-osd.0.asok config get osd_recovery_max_active 2024-09-16T05:25:33.851 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: local 'max_recoveries={"osd_recovery_max_active":"0"}' 2024-09-16T05:25:33.851 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:221: TEST_recovery_limit_adjustment_mclock: test '{"osd_recovery_max_active":"0"}' = '{"osd_recovery_max_active":"0"}' 2024-09-16T05:25:33.852 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:225: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_mclock_override_recovery_settings true 2024-09-16T05:25:34.371 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:226: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_recovery_max_active 10 2024-09-16T05:25:34.813 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:227: TEST_recovery_limit_adjustment_mclock: sleep 2 2024-09-16T05:25:36.818 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2024-09-16T05:25:36.818 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:25:36.818 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:25:36.818 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:25:36.818 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:25:36.818 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:25:36.818 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26942/ceph-osd.0.asok 2024-09-16T05:25:36.818 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2024-09-16T05:25:36.818 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26942/ceph-osd.0.asok config get osd_recovery_max_active 2024-09-16T05:25:36.902 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: max_recoveries='{"osd_recovery_max_active":"10"}' 2024-09-16T05:25:36.902 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:230: TEST_recovery_limit_adjustment_mclock: test '{"osd_recovery_max_active":"10"}' = '{"osd_recovery_max_active":"10"}' 2024-09-16T05:25:36.902 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:232: TEST_recovery_limit_adjustment_mclock: teardown td/mclock-config 2024-09-16T05:25:36.902 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T05:25:36.902 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:25:36.902 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T05:25:36.903 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:25:36.903 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:25:36.903 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:25:36.903 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:25:36.903 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:25:37.008 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:25:37.008 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:25:37.008 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:25:37.009 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:25:37.010 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:25:37.010 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:25:37.010 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:25:37.011 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:25:37.011 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:25:37.011 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:25:37.011 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:25:37.012 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:25:37.013 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:25:37.013 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T05:25:37.019 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:25:37.019 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:25:37.019 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:25:37.019 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26942 2024-09-16T05:25:37.020 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:25:37.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:25:37.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-09-16T05:25:37.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T05:25:37.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:25:37.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T05:25:37.021 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:25:37.021 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:25:37.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:25:37.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:25:37.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:25:37.022 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:25:37.022 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:25:37.023 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:25:37.023 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:25:37.024 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:25:37.024 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:25:37.024 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:25:37.025 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:25:37.025 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:25:37.025 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:25:37.025 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:25:37.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:25:37.027 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:25:37.027 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T05:25:37.028 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:25:37.028 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:25:37.028 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:25:37.028 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26942 2024-09-16T05:25:37.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:25:37.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:25:37.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-09-16T05:25:37.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/mclock-config 0 2024-09-16T05:25:37.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T05:25:37.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-16T05:25:37.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T05:25:37.030 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:25:37.030 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:25:37.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:25:37.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:25:37.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:25:37.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:25:37.031 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:25:37.032 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:25:37.032 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:25:37.033 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:25:37.033 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:25:37.033 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:25:37.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:25:37.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:25:37.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:25:37.035 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:25:37.035 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:25:37.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-16T05:25:37.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T05:25:37.037 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:25:37.037 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:25:37.037 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26942 2024-09-16T05:25:37.038 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26942 2024-09-16T05:25:37.038 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:25:37.038 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:25:37.038 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-09-16T05:25:37.040 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T05:25:37.040 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T05:25:37.106 INFO:tasks.workunit:Running workunit misc/network-ping.sh... 2024-09-16T05:25:37.106 DEBUG:teuthology.orchestra.run.smithi178:workunit test misc/network-ping.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ad35db06e8924912f01e32b35576297c0f1352b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh 2024-09-16T05:25:37.169 INFO:tasks.workunit.client.0.smithi178.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-16T05:25:37.174 INFO:tasks.workunit.client.0.smithi178.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-16T05:25:37.174 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-16T05:25:37.174 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-16T05:25:37.174 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-09-16T05:25:37.174 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-09-16T05:25:37.174 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-09-16T05:25:37.174 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-09-16T05:25:37.174 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-09-16T05:25:37.174 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-09-16T05:25:37.174 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/network-ping 2024-09-16T05:25:37.175 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:6: run: local dir=td/network-ping 2024-09-16T05:25:37.175 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:7: run: shift 2024-09-16T05:25:37.175 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:9: run: export CEPH_MON=127.0.0.1:7146 2024-09-16T05:25:37.175 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:9: run: CEPH_MON=127.0.0.1:7146 2024-09-16T05:25:37.175 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:10: run: export CEPH_ARGS 2024-09-16T05:25:37.175 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:11: run: uuidgen 2024-09-16T05:25:37.177 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:11: run: CEPH_ARGS+='--fsid=ad08e9ec-e980-42b8-a051-49ca1d64766e --auth-supported=none ' 2024-09-16T05:25:37.177 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:12: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-09-16T05:25:37.177 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:13: run: CEPH_ARGS+='--debug_disable_randomized_ping=true ' 2024-09-16T05:25:37.178 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:14: run: CEPH_ARGS+='--debug_heartbeat_testing_span=5 ' 2024-09-16T05:25:37.178 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:15: run: CEPH_ARGS+='--osd_heartbeat_interval=1 ' 2024-09-16T05:25:37.178 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: set 2024-09-16T05:25:37.178 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-16T05:25:37.178 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: local 'funcs=TEST_network_ping_test1 2024-09-16T05:25:37.178 INFO:tasks.workunit.client.0.smithi178.stderr:TEST_network_ping_test2' 2024-09-16T05:25:37.178 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:17: run: for func in $funcs 2024-09-16T05:25:37.178 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:18: run: setup td/network-ping 2024-09-16T05:25:37.178 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/network-ping 2024-09-16T05:25:37.178 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/network-ping 2024-09-16T05:25:37.178 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-09-16T05:25:37.178 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:25:37.178 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-09-16T05:25:37.178 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:25:37.179 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:25:37.179 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:25:37.179 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:25:37.179 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:25:37.179 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:25:37.180 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:25:37.180 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:25:37.180 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:25:37.180 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:25:37.181 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:25:37.181 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:25:37.182 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:25:37.182 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:25:37.182 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:25:37.182 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:25:37.182 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:25:37.183 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:25:37.183 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-09-16T05:25:37.184 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:25:37.184 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:25:37.184 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41120 2024-09-16T05:25:37.185 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.41120 2024-09-16T05:25:37.185 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:25:37.185 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:25:37.185 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/network-ping 2024-09-16T05:25:37.186 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:25:37.187 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:25:37.187 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41120 2024-09-16T05:25:37.187 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.41120 2024-09-16T05:25:37.189 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:25:37.189 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-16T05:25:37.189 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-16T05:25:37.189 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:25:37.189 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/network-ping 1' TERM HUP INT 2024-09-16T05:25:37.189 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:19: run: TEST_network_ping_test1 td/network-ping 2024-09-16T05:25:37.189 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:25: TEST_network_ping_test1: local dir=td/network-ping 2024-09-16T05:25:37.189 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:27: TEST_network_ping_test1: run_mon td/network-ping a 2024-09-16T05:25:37.189 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/network-ping 2024-09-16T05:25:37.189 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:25:37.189 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:25:37.189 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:25:37.189 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/network-ping/a 2024-09-16T05:25:37.190 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/network-ping/a --run-dir=td/network-ping 2024-09-16T05:25:37.586 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:25:37.587 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:25:37.587 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:25:37.587 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:25:37.587 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:25:37.587 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41120 2024-09-16T05:25:37.587 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.41120/$cluster-$name.asok' 2024-09-16T05:25:37.587 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/network-ping/a '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.41120/$cluster-$name.asok' --mon-cluster-log-file=td/network-ping/log --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T05:25:37.787 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:25:37.788 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:25:37.789 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:25:37.789 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:25:37.789 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:25:37.789 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:25:37.789 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:25:37.789 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:25:37.789 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:25:37.789 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:25:37.789 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:25:37.789 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41120 2024-09-16T05:25:37.789 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41120/ceph-mon.a.asok 2024-09-16T05:25:37.789 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:25:37.789 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.41120/ceph-mon.a.asok config get fsid 2024-09-16T05:25:37.871 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:25:37.872 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:25:37.872 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:25:37.872 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:25:37.872 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:25:37.872 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:25:37.872 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:25:37.872 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:25:37.872 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:25:37.872 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:25:37.872 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41120 2024-09-16T05:25:37.872 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41120/ceph-mon.a.asok 2024-09-16T05:25:37.872 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:25:37.873 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.41120/ceph-mon.a.asok config get mon_host 2024-09-16T05:25:37.950 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:28: TEST_network_ping_test1: run_mgr td/network-ping x 2024-09-16T05:25:37.950 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/network-ping 2024-09-16T05:25:37.950 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T05:25:37.950 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T05:25:37.950 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T05:25:37.950 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/network-ping/x 2024-09-16T05:25:37.950 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T05:25:38.791 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T05:25:38.791 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:25:38.791 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:25:38.791 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:25:38.792 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:25:38.792 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41120 2024-09-16T05:25:38.792 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.41120/$cluster-$name.asok' 2024-09-16T05:25:38.792 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:25:38.792 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/network-ping/x '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.41120/$cluster-$name.asok' --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:25:38.822 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:29: TEST_network_ping_test1: run_osd td/network-ping 0 2024-09-16T05:25:38.822 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-09-16T05:25:38.823 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:25:38.823 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T05:25:38.823 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:25:38.823 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/0 2024-09-16T05:25:38.823 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ad08e9ec-e980-42b8-a051-49ca1d64766e --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-09-16T05:25:38.823 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:25:38.823 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:25:38.823 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:25:38.823 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/0' 2024-09-16T05:25:38.823 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/0/journal' 2024-09-16T05:25:38.823 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:25:38.823 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:25:38.823 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-09-16T05:25:38.824 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:25:38.824 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:25:38.824 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:25:38.824 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:25:38.824 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:25:38.824 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41120 2024-09-16T05:25:38.824 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.41120/$cluster-$name.asok' 2024-09-16T05:25:38.824 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.41120/$cluster-$name.asok' 2024-09-16T05:25:38.824 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:25:38.824 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:25:38.824 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:25:38.824 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-09-16T05:25:38.824 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-09-16T05:25:38.824 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:25:38.825 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:25:38.825 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:25:38.825 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:25:38.825 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:25:38.825 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:25:38.825 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/0 2024-09-16T05:25:38.825 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:25:38.825 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=edd919a9-658f-4d86-8c2e-afe0116a72ba 2024-09-16T05:25:38.826 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 edd919a9-658f-4d86-8c2e-afe0116a72ba 2024-09-16T05:25:38.826 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 edd919a9-658f-4d86-8c2e-afe0116a72ba' 2024-09-16T05:25:38.826 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:25:38.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBSwedmVWQjMhAAAMUsLFpfCrPr6c3JvkZnJQ== 2024-09-16T05:25:38.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBSwedmVWQjMhAAAMUsLFpfCrPr6c3JvkZnJQ=="}' 2024-09-16T05:25:38.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new edd919a9-658f-4d86-8c2e-afe0116a72ba -i td/network-ping/0/new.json 2024-09-16T05:25:39.111 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:25:39.121 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/0/new.json 2024-09-16T05:25:39.122 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=ad08e9ec-e980-42b8-a051-49ca1d64766e --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41120/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBSwedmVWQjMhAAAMUsLFpfCrPr6c3JvkZnJQ== --osd-uuid edd919a9-658f-4d86-8c2e-afe0116a72ba 2024-09-16T05:25:39.151 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:25:39.150+0000 7f8f3caf1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:25:39.190 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:25:39.189+0000 7f8f3caf1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:25:39.191 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:25:39.191+0000 7f8f3caf1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:25:39.192 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:25:39.191+0000 7f8f3caf1640 -1 bdev(0x55ae74575c00 td/network-ping/0/block) open stat got: (1) Operation not permitted 2024-09-16T05:25:39.192 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:25:39.191+0000 7f8f3caf1640 -1 bluestore(td/network-ping/0) _read_fsid unparsable uuid 2024-09-16T05:25:44.706 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/0/keyring 2024-09-16T05:25:44.706 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:25:44.708 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T05:25:44.708 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T05:25:44.708 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:25:45.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T05:25:45.218 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:25:45.218 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=ad08e9ec-e980-42b8-a051-49ca1d64766e --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41120/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:25:45.218 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:25:45.218 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:25:45.218 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:25:45.245 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:25:45.244+0000 7fe44e214640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:25:45.270 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:25:45.269+0000 7fe44e214640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:25:45.271 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:25:45.270+0000 7fe44e214640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:25:45.666 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T05:25:45.666 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:25:45.666 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:25:45.666 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:25:45.666 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:25:45.666 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:25:45.667 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:25:45.667 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:25:45.667 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:25:45.667 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:25:46.083 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:25:46.627 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:25:46.626+0000 7fe44e214640 -1 Falling back to public interface 2024-09-16T05:25:47.085 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:25:47.085 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:25:47.085 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:25:47.086 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:25:47.086 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:25:47.086 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:25:47.457 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:25:48.459 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:25:48.459 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:25:48.459 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:25:48.460 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:25:48.460 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:25:48.460 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:25:48.841 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:25:49.406 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:25:49.404+0000 7fe44e214640 -1 osd.0 0 log_to_monitors true 2024-09-16T05:25:49.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:25:49.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:25:49.842 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:25:49.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:25:49.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:25:49.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:25:50.310 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:25:51.311 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:25:51.311 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:25:51.311 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:25:51.312 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:25:51.312 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:25:51.312 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:25:51.690 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:25:52.691 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:25:52.691 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:25:52.691 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:25:52.691 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:25:52.692 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:25:52.692 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:25:53.112 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:25:54.113 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:25:54.113 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:25:54.113 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:25:54.114 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:25:54.114 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:25:54.114 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:25:54.509 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:25:55.510 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:25:55.511 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:25:55.511 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:25:55.511 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:25:55.511 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:25:55.512 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:25:55.914 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:25:56.915 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:25:56.915 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:25:56.916 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:25:56.916 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:25:56.916 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:25:56.916 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:25:57.297 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:25:58.298 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:25:58.298 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:25:58.298 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:25:58.299 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:25:58.299 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:25:58.299 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:25:58.684 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:25:59.686 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:25:59.686 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:25:59.686 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:25:59.687 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:25:59.687 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:25:59.687 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:26:00.126 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:26:01.127 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:26:01.127 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:26:01.128 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:26:01.128 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:26:01.128 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:26:01.128 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:26:01.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:26:02.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:26:02.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:26:02.844 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:26:02.844 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:26:02.844 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:26:02.844 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:26:03.866 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:26:04.868 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:26:04.884 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:26:04.884 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:26:04.885 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:26:04.885 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:26:04.885 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:26:05.645 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:26:06.646 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:26:06.646 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:26:06.646 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:26:06.647 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:26:06.647 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:26:06.647 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:26:07.499 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:26:08.500 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:26:08.500 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:26:08.500 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:26:08.501 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:26:08.501 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:26:08.501 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:26:09.331 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:26:10.332 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:26:10.332 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:26:10.333 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:26:10.333 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:26:10.333 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:26:10.333 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:26:10.990 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:26:11.991 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:26:11.991 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:26:11.992 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:26:11.992 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:26:11.992 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:26:11.992 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:26:12.414 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3330560327,v1:127.0.0.1:6803/3330560327] [v2:127.0.0.1:6804/3330560327,v1:127.0.0.1:6805/3330560327] exists,up edd919a9-658f-4d86-8c2e-afe0116a72ba 2024-09-16T05:26:12.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:26:12.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:26:12.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:26:12.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:30: TEST_network_ping_test1: run_osd td/network-ping 1 2024-09-16T05:26:12.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-09-16T05:26:12.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:26:12.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T05:26:12.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:26:12.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/1 2024-09-16T05:26:12.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ad08e9ec-e980-42b8-a051-49ca1d64766e --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-09-16T05:26:12.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:26:12.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:26:12.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:26:12.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/1' 2024-09-16T05:26:12.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/1/journal' 2024-09-16T05:26:12.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:26:12.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:26:12.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-09-16T05:26:12.419 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:26:12.420 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:26:12.420 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:26:12.420 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:26:12.420 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:26:12.420 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41120 2024-09-16T05:26:12.420 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.41120/$cluster-$name.asok' 2024-09-16T05:26:12.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.41120/$cluster-$name.asok' 2024-09-16T05:26:12.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:26:12.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:26:12.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:26:12.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-09-16T05:26:12.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-09-16T05:26:12.421 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:26:12.421 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:26:12.421 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:26:12.421 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:26:12.421 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:26:12.421 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:26:12.421 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/1 2024-09-16T05:26:12.421 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:26:12.421 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ae4ed3c3-de23-4411-af75-148479aa594f 2024-09-16T05:26:12.421 INFO:tasks.workunit.client.0.smithi178.stdout:add osd1 ae4ed3c3-de23-4411-af75-148479aa594f 2024-09-16T05:26:12.422 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 ae4ed3c3-de23-4411-af75-148479aa594f' 2024-09-16T05:26:12.422 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:26:12.436 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB0wedmTlv2GRAAQkyylnMgX4+DyraRi82reA== 2024-09-16T05:26:12.436 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB0wedmTlv2GRAAQkyylnMgX4+DyraRi82reA=="}' 2024-09-16T05:26:12.436 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ae4ed3c3-de23-4411-af75-148479aa594f -i td/network-ping/1/new.json 2024-09-16T05:26:12.936 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:26:12.948 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/1/new.json 2024-09-16T05:26:12.948 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=ad08e9ec-e980-42b8-a051-49ca1d64766e --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41120/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB0wedmTlv2GRAAQkyylnMgX4+DyraRi82reA== --osd-uuid ae4ed3c3-de23-4411-af75-148479aa594f 2024-09-16T05:26:12.978 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:26:12.977+0000 7fdf2ce9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:26:12.996 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:26:12.995+0000 7fdf2ce9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:26:12.997 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:26:12.997+0000 7fdf2ce9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:26:12.998 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:26:12.997+0000 7fdf2ce9c640 -1 bdev(0x55d8de21c800 td/network-ping/1/block) open stat got: (1) Operation not permitted 2024-09-16T05:26:12.998 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:26:12.997+0000 7fdf2ce9c640 -1 bluestore(td/network-ping/1) _read_fsid unparsable uuid 2024-09-16T05:26:18.860 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/1/keyring 2024-09-16T05:26:18.860 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:26:18.861 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd1 key to auth repository 2024-09-16T05:26:18.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T05:26:18.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:26:19.384 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.1 2024-09-16T05:26:19.384 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T05:26:19.384 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=ad08e9ec-e980-42b8-a051-49ca1d64766e --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41120/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:26:19.385 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:26:19.385 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:26:19.385 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:26:19.412 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:26:19.411+0000 7f62dde15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:26:19.435 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:26:19.434+0000 7f62dde15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:26:19.436 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:26:19.435+0000 7f62dde15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:26:19.799 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T05:26:19.799 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:26:19.799 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T05:26:19.799 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:26:19.799 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:26:19.799 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:26:19.800 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:26:19.800 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:26:19.800 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:26:19.800 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:26:20.216 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:26:21.034 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:26:21.034+0000 7f62dde15640 -1 Falling back to public interface 2024-09-16T05:26:21.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:26:21.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:26:21.218 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:26:21.218 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:26:21.218 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:26:21.218 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:26:21.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:26:22.594 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:26:22.594 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:26:22.594 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:26:22.594 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:26:22.595 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:26:22.595 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:26:22.990 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:26:23.923 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:26:23.922+0000 7f62dde15640 -1 osd.1 0 log_to_monitors true 2024-09-16T05:26:23.992 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:26:23.992 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:26:23.992 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:26:23.992 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:26:23.992 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:26:23.993 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:26:24.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:26:25.594 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:26:25.594 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:26:25.594 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:26:25.595 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:26:25.595 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:26:25.595 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:26:25.979 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:26:26.980 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:26:26.981 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:26:26.981 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:26:26.982 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:26:26.982 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:26:26.983 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:26:27.386 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:26:28.388 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:26:28.388 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:26:28.388 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:26:28.388 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:26:28.389 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:26:28.389 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:26:28.826 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:26:29.827 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:26:29.827 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:26:29.827 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:26:29.828 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:26:29.828 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:26:29.828 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:26:30.210 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:26:31.211 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:26:31.211 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:26:31.211 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:26:31.212 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:26:31.212 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:26:31.212 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:26:31.667 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:26:32.668 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:26:32.668 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:26:32.669 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:26:32.669 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:26:32.669 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:26:32.669 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:26:33.193 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:26:34.194 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:26:34.194 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:26:34.194 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:26:34.195 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:26:34.195 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:26:34.195 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:26:34.746 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:26:35.747 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:26:35.747 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:26:35.748 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:26:35.748 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:26:35.748 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:26:35.748 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:26:36.359 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:26:37.360 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:26:37.360 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:26:37.360 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:26:37.361 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:26:37.361 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:26:37.361 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:26:38.181 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:26:39.182 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:26:39.182 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:26:39.183 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:26:39.183 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:26:39.183 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:26:39.183 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:26:40.191 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:26:41.192 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:26:41.193 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:26:41.193 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:26:41.193 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:26:41.193 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:26:41.194 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:26:41.919 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:26:42.921 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:26:42.921 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:26:42.921 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:26:42.922 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:26:42.923 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:26:42.923 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:26:43.697 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:26:44.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:26:44.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:26:44.699 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:26:44.699 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:26:44.699 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:26:44.699 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:26:45.792 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:26:46.793 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:26:46.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:26:46.794 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:26:46.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:26:46.795 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:26:46.795 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:26:47.180 INFO:tasks.workunit.client.0.smithi178.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1664448498,v1:127.0.0.1:6811/1664448498] [v2:127.0.0.1:6812/1664448498,v1:127.0.0.1:6813/1664448498] exists,up ae4ed3c3-de23-4411-af75-148479aa594f 2024-09-16T05:26:47.185 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:26:47.185 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:26:47.185 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:26:47.185 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:31: TEST_network_ping_test1: run_osd td/network-ping 2 2024-09-16T05:26:47.185 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-09-16T05:26:47.185 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:26:47.185 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T05:26:47.186 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:26:47.186 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/2 2024-09-16T05:26:47.186 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ad08e9ec-e980-42b8-a051-49ca1d64766e --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-09-16T05:26:47.186 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:26:47.186 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:26:47.186 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:26:47.186 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/2' 2024-09-16T05:26:47.186 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/2/journal' 2024-09-16T05:26:47.186 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:26:47.186 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:26:47.186 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-09-16T05:26:47.187 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:26:47.187 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:26:47.187 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:26:47.187 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:26:47.187 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:26:47.187 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41120 2024-09-16T05:26:47.187 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.41120/$cluster-$name.asok' 2024-09-16T05:26:47.187 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.41120/$cluster-$name.asok' 2024-09-16T05:26:47.187 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:26:47.187 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:26:47.187 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:26:47.187 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-09-16T05:26:47.187 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-09-16T05:26:47.188 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:26:47.188 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:26:47.188 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:26:47.188 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:26:47.188 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:26:47.188 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:26:47.188 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/2 2024-09-16T05:26:47.188 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:26:47.188 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7ad0de49-cba1-40d8-937c-dc3a862f2690 2024-09-16T05:26:47.188 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 7ad0de49-cba1-40d8-937c-dc3a862f2690' 2024-09-16T05:26:47.188 INFO:tasks.workunit.client.0.smithi178.stdout:add osd2 7ad0de49-cba1-40d8-937c-dc3a862f2690 2024-09-16T05:26:47.189 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:26:47.203 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCXwedme1sLDBAAYGhlwjiqO4GNcP61iYm28w== 2024-09-16T05:26:47.203 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCXwedme1sLDBAAYGhlwjiqO4GNcP61iYm28w=="}' 2024-09-16T05:26:47.203 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7ad0de49-cba1-40d8-937c-dc3a862f2690 -i td/network-ping/2/new.json 2024-09-16T05:26:47.633 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:26:47.644 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/2/new.json 2024-09-16T05:26:47.645 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=ad08e9ec-e980-42b8-a051-49ca1d64766e --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41120/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCXwedme1sLDBAAYGhlwjiqO4GNcP61iYm28w== --osd-uuid 7ad0de49-cba1-40d8-937c-dc3a862f2690 2024-09-16T05:26:47.674 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:26:47.672+0000 7f780d016640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:26:47.680 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:26:47.678+0000 7f780d016640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:26:47.681 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:26:47.679+0000 7f780d016640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:26:47.681 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:26:47.680+0000 7f780d016640 -1 bdev(0x55a7c88dfc00 td/network-ping/2/block) open stat got: (1) Operation not permitted 2024-09-16T05:26:47.681 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:26:47.680+0000 7f780d016640 -1 bluestore(td/network-ping/2) _read_fsid unparsable uuid 2024-09-16T05:26:53.335 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/2/keyring 2024-09-16T05:26:53.335 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:26:53.336 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd2 key to auth repository 2024-09-16T05:26:53.336 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T05:26:53.336 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:26:53.849 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.2 2024-09-16T05:26:53.850 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T05:26:53.850 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=ad08e9ec-e980-42b8-a051-49ca1d64766e --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41120/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:26:53.850 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:26:53.850 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:26:53.850 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:26:53.877 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:26:53.876+0000 7ffbf81b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:26:53.894 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:26:53.893+0000 7ffbf81b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:26:53.895 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:26:53.894+0000 7ffbf81b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:26:54.301 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T05:26:54.301 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:26:54.301 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T05:26:54.301 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:26:54.302 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:26:54.302 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:26:54.302 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:26:54.302 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:26:54.302 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:26:54.303 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:26:54.708 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:26:55.710 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:26:55.710 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:26:55.710 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:26:55.711 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:26:55.711 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:26:55.711 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:26:55.751 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:26:55.750+0000 7ffbf81b2640 -1 Falling back to public interface 2024-09-16T05:26:56.161 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:26:57.162 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:26:57.162 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:26:57.163 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:26:57.163 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:26:57.163 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:26:57.163 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:26:57.579 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:26:58.400 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:26:58.399+0000 7ffbf81b2640 -1 osd.2 0 log_to_monitors true 2024-09-16T05:26:58.580 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:26:58.581 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:26:58.581 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:26:58.581 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:26:58.582 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:26:58.582 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:26:59.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:27:00.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:27:00.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:27:00.044 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:27:00.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:27:00.045 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:27:00.045 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:27:00.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:27:01.533 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:27:01.533 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:27:01.534 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:27:01.534 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:27:01.535 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:27:01.535 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:27:01.922 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:27:02.923 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:27:02.923 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:27:02.923 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:27:02.924 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:27:02.924 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:27:02.924 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:27:03.300 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:27:04.300 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:27:04.300 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:27:04.300 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:27:04.301 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:27:04.301 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:27:04.301 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:27:04.678 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:27:05.679 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:27:05.679 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:27:05.679 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:27:05.680 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:27:05.680 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:27:05.680 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:27:06.210 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:27:07.211 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:27:07.211 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:27:07.212 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:27:07.212 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:27:07.212 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:27:07.212 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:27:07.623 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:27:08.625 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:27:08.625 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:27:08.625 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:27:08.625 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:27:08.626 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:27:08.626 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:27:09.183 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:27:10.184 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:27:10.185 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:27:10.185 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:27:10.185 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:27:10.186 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:27:10.186 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:27:10.776 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:27:11.777 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:27:11.777 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:27:11.778 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:27:11.778 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:27:11.778 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:27:11.778 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:27:12.590 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:27:13.591 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:27:13.591 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:27:13.592 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:27:13.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:27:13.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:27:13.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:27:14.339 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:27:15.340 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:27:15.340 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:27:15.341 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:27:15.341 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:27:15.341 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:27:15.341 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:27:16.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:27:17.396 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:27:17.396 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:27:17.396 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:27:17.397 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:27:17.397 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:27:17.397 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:27:18.079 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:27:19.084 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:27:19.085 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:27:19.085 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:27:19.086 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:27:19.086 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:27:19.086 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:27:19.849 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:27:20.850 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:27:20.851 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:27:20.851 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:27:20.851 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:27:20.851 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:27:20.852 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:27:21.236 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:27:22.237 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:27:22.237 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:27:22.238 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T05:27:22.238 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T05:27:22.238 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:27:22.238 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:27:22.664 INFO:tasks.workunit.client.0.smithi178.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2239713195,v1:127.0.0.1:6819/2239713195] [v2:127.0.0.1:6820/2239713195,v1:127.0.0.1:6821/2239713195] exists,up 7ad0de49-cba1-40d8-937c-dc3a862f2690 2024-09-16T05:27:22.665 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:27:22.665 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:27:22.665 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:27:22.665 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:33: TEST_network_ping_test1: sleep 5 2024-09-16T05:27:27.666 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:35: TEST_network_ping_test1: create_pool foo 16 2024-09-16T05:27:27.666 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-09-16T05:27:28.143 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'foo' already exists 2024-09-16T05:27:28.155 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T05:27:29.156 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:38: TEST_network_ping_test1: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2024-09-16T05:27:29.200 INFO:tasks.workunit.client.0.smithi178.stdout:hints = 1 2024-09-16T05:27:29.200 INFO:tasks.workunit.client.0.smithi178.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2024-09-16T05:27:29.200 INFO:tasks.workunit.client.0.smithi178.stdout:Object prefix: benchmark_data_smithi178_46670 2024-09-16T05:27:29.200 INFO:tasks.workunit.client.0.smithi178.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T05:27:29.200 INFO:tasks.workunit.client.0.smithi178.stdout: 0 0 0 0 0 0 - 0 2024-09-16T05:27:30.200 INFO:tasks.workunit.client.0.smithi178.stdout: 1 16 72 56 0.218738 0.21875 0.237264 0.266369 2024-09-16T05:27:31.200 INFO:tasks.workunit.client.0.smithi178.stdout: 2 16 136 120 0.234358 0.25 0.222997 0.248372 2024-09-16T05:27:32.200 INFO:tasks.workunit.client.0.smithi178.stdout: 3 16 206 190 0.247376 0.273438 0.234945 0.24075 2024-09-16T05:27:33.200 INFO:tasks.workunit.client.0.smithi178.stdout: 4 16 270 254 0.248024 0.25 0.242149 0.242953 2024-09-16T05:27:34.200 INFO:tasks.workunit.client.0.smithi178.stdout: 5 16 344 328 0.256225 0.289062 0.258922 0.239968 2024-09-16T05:27:35.200 INFO:tasks.workunit.client.0.smithi178.stdout: 6 16 412 396 0.257785 0.265625 0.220981 0.238597 2024-09-16T05:27:36.201 INFO:tasks.workunit.client.0.smithi178.stdout: 7 16 481 465 0.259458 0.269531 0.218175 0.235704 2024-09-16T05:27:37.201 INFO:tasks.workunit.client.0.smithi178.stdout: 8 16 545 529 0.258273 0.25 0.300898 0.235965 2024-09-16T05:27:38.201 INFO:tasks.workunit.client.0.smithi178.stdout: 9 16 625 609 0.264293 0.3125 0.142874 0.235669 2024-09-16T05:27:39.201 INFO:tasks.workunit.client.0.smithi178.stdout: 10 16 705 689 0.269109 0.3125 0.168063 0.231776 2024-09-16T05:27:40.201 INFO:tasks.workunit.client.0.smithi178.stdout:Total time run: 10.1585 2024-09-16T05:27:40.201 INFO:tasks.workunit.client.0.smithi178.stdout:Total writes made: 705 2024-09-16T05:27:40.201 INFO:tasks.workunit.client.0.smithi178.stdout:Write size: 4096 2024-09-16T05:27:40.201 INFO:tasks.workunit.client.0.smithi178.stdout:Object size: 4096 2024-09-16T05:27:40.201 INFO:tasks.workunit.client.0.smithi178.stdout:Bandwidth (MB/sec): 0.271094 2024-09-16T05:27:40.201 INFO:tasks.workunit.client.0.smithi178.stdout:Stddev Bandwidth: 0.0294887 2024-09-16T05:27:40.202 INFO:tasks.workunit.client.0.smithi178.stdout:Max bandwidth (MB/sec): 0.3125 2024-09-16T05:27:40.202 INFO:tasks.workunit.client.0.smithi178.stdout:Min bandwidth (MB/sec): 0.21875 2024-09-16T05:27:40.202 INFO:tasks.workunit.client.0.smithi178.stdout:Average IOPS: 69 2024-09-16T05:27:40.202 INFO:tasks.workunit.client.0.smithi178.stdout:Stddev IOPS: 7.5491 2024-09-16T05:27:40.202 INFO:tasks.workunit.client.0.smithi178.stdout:Max IOPS: 80 2024-09-16T05:27:40.202 INFO:tasks.workunit.client.0.smithi178.stdout:Min IOPS: 56 2024-09-16T05:27:40.202 INFO:tasks.workunit.client.0.smithi178.stdout:Average Latency(s): 0.230296 2024-09-16T05:27:40.202 INFO:tasks.workunit.client.0.smithi178.stdout:Stddev Latency(s): 0.0319523 2024-09-16T05:27:40.202 INFO:tasks.workunit.client.0.smithi178.stdout:Max latency(s): 0.388855 2024-09-16T05:27:40.202 INFO:tasks.workunit.client.0.smithi178.stdout:Min latency(s): 0.127256 2024-09-16T05:27:40.321 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:41: TEST_network_ping_test1: sleep 10 2024-09-16T05:27:50.323 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:42: TEST_network_ping_test1: flush_pg_stats 2024-09-16T05:27:50.324 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-16T05:27:50.324 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-16T05:27:50.761 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-16T05:27:50.762 INFO:tasks.workunit.client.0.smithi178.stderr:1 2024-09-16T05:27:50.762 INFO:tasks.workunit.client.0.smithi178.stderr:2' 2024-09-16T05:27:50.762 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-16T05:27:50.762 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:27:50.762 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T05:27:50.930 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836503 2024-09-16T05:27:50.930 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836503 2024-09-16T05:27:50.930 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503' 2024-09-16T05:27:50.930 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:27:50.930 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T05:27:51.063 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672980 2024-09-16T05:27:51.064 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672980 2024-09-16T05:27:51.064 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672980' 2024-09-16T05:27:51.064 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:27:51.064 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T05:27:51.175 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-09-16T05:27:51.175 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-09-16T05:27:51.176 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672980 2-64424509453' 2024-09-16T05:27:51.176 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:27:51.176 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836503 2024-09-16T05:27:51.176 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:27:51.177 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-16T05:27:51.177 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836503 2024-09-16T05:27:51.177 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:27:51.178 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.0 seq 21474836503 2024-09-16T05:27:51.179 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836503 2024-09-16T05:27:51.179 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-09-16T05:27:51.179 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:27:51.588 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836503 2024-09-16T05:27:51.589 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-16T05:27:52.590 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T05:27:52.590 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:27:52.972 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836503 2024-09-16T05:27:52.972 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:27:52.973 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672980 2024-09-16T05:27:52.973 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:27:52.973 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-09-16T05:27:52.974 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672980 2024-09-16T05:27:52.974 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:27:52.975 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672980 2024-09-16T05:27:52.975 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.1 seq 42949672980 2024-09-16T05:27:52.976 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-09-16T05:27:52.976 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T05:27:53.367 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672981 -lt 42949672980 2024-09-16T05:27:53.367 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:27:53.367 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-09-16T05:27:53.367 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:27:53.368 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-09-16T05:27:53.369 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-09-16T05:27:53.369 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:27:53.370 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-09-16T05:27:53.370 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.2 seq 64424509453 2024-09-16T05:27:53.370 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-09-16T05:27:53.370 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T05:27:53.738 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509453 2024-09-16T05:27:53.739 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: tee td/network-ping/json 2024-09-16T05:27:53.740 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: get_asok_path osd.0 2024-09-16T05:27:53.740 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:27:53.740 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:27:53.740 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:27:53.740 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:27:53.740 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41120 2024-09-16T05:27:53.740 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41120/ceph-osd.0.asok 2024-09-16T05:27:53.741 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: CEPH_ARGS= 2024-09-16T05:27:53.741 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41120/ceph-osd.0.asok dump_osd_network 2024-09-16T05:27:53.817 INFO:tasks.workunit.client.0.smithi178.stdout:{ 2024-09-16T05:27:53.818 INFO:tasks.workunit.client.0.smithi178.stdout: "threshold": 1000, 2024-09-16T05:27:53.818 INFO:tasks.workunit.client.0.smithi178.stdout: "entries": [] 2024-09-16T05:27:53.818 INFO:tasks.workunit.client.0.smithi178.stdout:} 2024-09-16T05:27:53.825 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T05:27:53.825 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: jq '.entries | length' 2024-09-16T05:27:53.827 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: test 0 = 0 2024-09-16T05:27:53.828 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T05:27:53.828 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: jq .threshold 2024-09-16T05:27:53.830 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: test 1000 = 1000 2024-09-16T05:27:53.831 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: tee td/network-ping/json 2024-09-16T05:27:53.832 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-16T05:27:53.832 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-09-16T05:27:53.832 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-09-16T05:27:53.832 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:27:53.832 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:27:53.833 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41120 2024-09-16T05:27:53.833 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41120/ceph-mgr.x.asok 2024-09-16T05:27:53.833 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: CEPH_ARGS= 2024-09-16T05:27:53.833 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41120/ceph-mgr.x.asok dump_osd_network 2024-09-16T05:27:53.902 INFO:tasks.workunit.client.0.smithi178.stdout:{ 2024-09-16T05:27:53.903 INFO:tasks.workunit.client.0.smithi178.stdout: "threshold": 1000, 2024-09-16T05:27:53.903 INFO:tasks.workunit.client.0.smithi178.stdout: "entries": [] 2024-09-16T05:27:53.903 INFO:tasks.workunit.client.0.smithi178.stdout:} 2024-09-16T05:27:53.910 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T05:27:53.910 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: jq '.entries | length' 2024-09-16T05:27:53.913 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: test 0 = 0 2024-09-16T05:27:53.913 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T05:27:53.913 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: jq .threshold 2024-09-16T05:27:53.916 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: test 1000 = 1000 2024-09-16T05:27:53.917 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: tee td/network-ping/json 2024-09-16T05:27:53.917 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: get_asok_path osd.0 2024-09-16T05:27:53.917 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:27:53.917 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:27:53.917 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:27:53.917 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:27:53.917 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41120 2024-09-16T05:27:53.918 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41120/ceph-osd.0.asok 2024-09-16T05:27:53.918 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: CEPH_ARGS= 2024-09-16T05:27:53.918 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41120/ceph-osd.0.asok dump_osd_network 0 2024-09-16T05:27:53.994 INFO:tasks.workunit.client.0.smithi178.stdout:{ 2024-09-16T05:27:53.994 INFO:tasks.workunit.client.0.smithi178.stdout: "threshold": 0, 2024-09-16T05:27:53.994 INFO:tasks.workunit.client.0.smithi178.stdout: "entries": [ 2024-09-16T05:27:53.994 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:27:53.995 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:27:53 2024", 2024-09-16T05:27:53.995 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:27:53.995 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 0, 2024-09-16T05:27:53.995 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 2, 2024-09-16T05:27:53.995 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "front", 2024-09-16T05:27:53.995 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:27:53.995 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.299, 2024-09-16T05:27:53.995 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.341, 2024-09-16T05:27:53.995 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.338 2024-09-16T05:27:53.995 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:53.995 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:27:53.995 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.401, 2024-09-16T05:27:53.996 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.572, 2024-09-16T05:27:53.996 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.572 2024-09-16T05:27:53.996 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:53.996 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:27:53.996 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.401, 2024-09-16T05:27:53.996 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.572, 2024-09-16T05:27:53.996 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.572 2024-09-16T05:27:53.996 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:53.996 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.206 2024-09-16T05:27:53.996 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:53.996 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:27:53.996 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:27:53 2024", 2024-09-16T05:27:53.997 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:27:53.997 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 0, 2024-09-16T05:27:53.997 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 2, 2024-09-16T05:27:53.997 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "back", 2024-09-16T05:27:53.997 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:27:53.997 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.269, 2024-09-16T05:27:53.997 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.313, 2024-09-16T05:27:53.997 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.324 2024-09-16T05:27:53.997 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:53.997 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:27:53.997 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.378, 2024-09-16T05:27:53.997 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.462, 2024-09-16T05:27:53.997 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.462 2024-09-16T05:27:53.997 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:53.998 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:27:53.998 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.378, 2024-09-16T05:27:53.998 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.462, 2024-09-16T05:27:53.998 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.462 2024-09-16T05:27:53.998 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:53.998 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.250 2024-09-16T05:27:53.998 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:53.998 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:27:53.998 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:27:52 2024", 2024-09-16T05:27:53.998 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:27:53.998 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 0, 2024-09-16T05:27:53.998 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 1, 2024-09-16T05:27:53.998 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "front", 2024-09-16T05:27:53.998 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:27:53.999 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.299, 2024-09-16T05:27:53.999 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.305, 2024-09-16T05:27:53.999 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.294 2024-09-16T05:27:53.999 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:53.999 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:27:53.999 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.380, 2024-09-16T05:27:53.999 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.517, 2024-09-16T05:27:53.999 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.517 2024-09-16T05:27:53.999 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:53.999 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:27:53.999 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.380, 2024-09-16T05:27:53.999 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.517, 2024-09-16T05:27:53.999 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.517 2024-09-16T05:27:54.000 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.000 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.231 2024-09-16T05:27:54.000 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.000 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:27:54.000 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:27:52 2024", 2024-09-16T05:27:54.000 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:27:54.000 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 0, 2024-09-16T05:27:54.000 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 1, 2024-09-16T05:27:54.000 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "back", 2024-09-16T05:27:54.000 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:27:54.001 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.263, 2024-09-16T05:27:54.001 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.290, 2024-09-16T05:27:54.001 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.281 2024-09-16T05:27:54.001 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.001 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:27:54.001 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.350, 2024-09-16T05:27:54.001 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.498, 2024-09-16T05:27:54.002 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.498 2024-09-16T05:27:54.002 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.002 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:27:54.002 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.350, 2024-09-16T05:27:54.002 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.498, 2024-09-16T05:27:54.002 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.498 2024-09-16T05:27:54.002 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.002 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.219 2024-09-16T05:27:54.002 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T05:27:54.002 INFO:tasks.workunit.client.0.smithi178.stdout: ] 2024-09-16T05:27:54.002 INFO:tasks.workunit.client.0.smithi178.stdout:} 2024-09-16T05:27:54.003 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T05:27:54.003 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: jq '.entries | length' 2024-09-16T05:27:54.004 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: test 4 = 4 2024-09-16T05:27:54.005 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T05:27:54.005 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: jq .threshold 2024-09-16T05:27:54.007 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: test 0 = 0 2024-09-16T05:27:54.009 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: tee td/network-ping/json 2024-09-16T05:27:54.009 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-16T05:27:54.009 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-09-16T05:27:54.009 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-09-16T05:27:54.009 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:27:54.009 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:27:54.009 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41120 2024-09-16T05:27:54.009 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41120/ceph-mgr.x.asok 2024-09-16T05:27:54.009 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: CEPH_ARGS= 2024-09-16T05:27:54.009 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41120/ceph-mgr.x.asok dump_osd_network 0 2024-09-16T05:27:54.080 INFO:tasks.workunit.client.0.smithi178.stdout:{ 2024-09-16T05:27:54.081 INFO:tasks.workunit.client.0.smithi178.stdout: "threshold": 0, 2024-09-16T05:27:54.081 INFO:tasks.workunit.client.0.smithi178.stdout: "entries": [ 2024-09-16T05:27:54.081 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:27:54.081 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:27:47 2024", 2024-09-16T05:27:54.081 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:27:54.081 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 2, 2024-09-16T05:27:54.081 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 1, 2024-09-16T05:27:54.081 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "back", 2024-09-16T05:27:54.082 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:27:54.082 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.362, 2024-09-16T05:27:54.082 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.326, 2024-09-16T05:27:54.082 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.331 2024-09-16T05:27:54.082 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.082 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:27:54.082 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.251, 2024-09-16T05:27:54.082 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.215, 2024-09-16T05:27:54.082 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.215 2024-09-16T05:27:54.082 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.082 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:27:54.082 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.400, 2024-09-16T05:27:54.082 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.434, 2024-09-16T05:27:54.082 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.434 2024-09-16T05:27:54.082 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.083 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.252 2024-09-16T05:27:54.083 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.083 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:27:54.083 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:27:47 2024", 2024-09-16T05:27:54.083 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:27:54.083 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 2, 2024-09-16T05:27:54.083 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 1, 2024-09-16T05:27:54.083 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "front", 2024-09-16T05:27:54.083 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:27:54.083 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.352, 2024-09-16T05:27:54.083 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.335, 2024-09-16T05:27:54.083 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.342 2024-09-16T05:27:54.083 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.084 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:27:54.084 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.267, 2024-09-16T05:27:54.084 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.236, 2024-09-16T05:27:54.084 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.236 2024-09-16T05:27:54.084 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.084 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:27:54.084 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.415, 2024-09-16T05:27:54.084 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.419, 2024-09-16T05:27:54.084 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.419 2024-09-16T05:27:54.084 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.084 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.266 2024-09-16T05:27:54.084 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.084 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:27:54.084 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:27:48 2024", 2024-09-16T05:27:54.085 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:27:54.085 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 0, 2024-09-16T05:27:54.085 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 2, 2024-09-16T05:27:54.085 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "front", 2024-09-16T05:27:54.085 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:27:54.085 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.345, 2024-09-16T05:27:54.085 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.349, 2024-09-16T05:27:54.085 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.341 2024-09-16T05:27:54.085 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.085 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:27:54.085 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.286, 2024-09-16T05:27:54.085 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.222, 2024-09-16T05:27:54.085 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.222 2024-09-16T05:27:54.086 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.086 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:27:54.086 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.428, 2024-09-16T05:27:54.086 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.572, 2024-09-16T05:27:54.086 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.572 2024-09-16T05:27:54.086 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.086 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.275 2024-09-16T05:27:54.086 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.086 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:27:54.086 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:27:48 2024", 2024-09-16T05:27:54.086 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:27:54.086 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 0, 2024-09-16T05:27:54.086 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 2, 2024-09-16T05:27:54.086 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "back", 2024-09-16T05:27:54.087 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:27:54.087 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.320, 2024-09-16T05:27:54.087 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.325, 2024-09-16T05:27:54.087 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.328 2024-09-16T05:27:54.087 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.087 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:27:54.087 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.270, 2024-09-16T05:27:54.087 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.232, 2024-09-16T05:27:54.087 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.232 2024-09-16T05:27:54.087 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.087 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:27:54.087 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.412, 2024-09-16T05:27:54.087 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.462, 2024-09-16T05:27:54.087 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.462 2024-09-16T05:27:54.087 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.087 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.224 2024-09-16T05:27:54.088 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.088 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:27:54.088 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:27:47 2024", 2024-09-16T05:27:54.088 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:27:54.088 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 2, 2024-09-16T05:27:54.088 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 0, 2024-09-16T05:27:54.088 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "front", 2024-09-16T05:27:54.088 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:27:54.088 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.327, 2024-09-16T05:27:54.088 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.313, 2024-09-16T05:27:54.088 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.319 2024-09-16T05:27:54.089 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.089 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:27:54.089 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.213, 2024-09-16T05:27:54.089 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.194, 2024-09-16T05:27:54.089 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.194 2024-09-16T05:27:54.089 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.089 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:27:54.089 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.377, 2024-09-16T05:27:54.089 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.411, 2024-09-16T05:27:54.089 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.411 2024-09-16T05:27:54.089 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.089 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.233 2024-09-16T05:27:54.090 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.090 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:27:54.090 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:27:47 2024", 2024-09-16T05:27:54.090 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:27:54.090 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 2, 2024-09-16T05:27:54.090 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 0, 2024-09-16T05:27:54.090 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "back", 2024-09-16T05:27:54.090 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:27:54.090 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.325, 2024-09-16T05:27:54.090 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.308, 2024-09-16T05:27:54.090 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.318 2024-09-16T05:27:54.090 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.090 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:27:54.090 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.226, 2024-09-16T05:27:54.090 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.226, 2024-09-16T05:27:54.091 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.226 2024-09-16T05:27:54.091 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.091 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:27:54.091 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.455, 2024-09-16T05:27:54.091 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.455, 2024-09-16T05:27:54.091 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.455 2024-09-16T05:27:54.091 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.091 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.289 2024-09-16T05:27:54.091 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.091 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:27:54.091 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:27:47 2024", 2024-09-16T05:27:54.092 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:27:54.092 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 1, 2024-09-16T05:27:54.092 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 2, 2024-09-16T05:27:54.092 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "front", 2024-09-16T05:27:54.092 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:27:54.092 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.322, 2024-09-16T05:27:54.092 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.320, 2024-09-16T05:27:54.092 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.304 2024-09-16T05:27:54.092 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.092 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:27:54.092 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.266, 2024-09-16T05:27:54.092 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.213, 2024-09-16T05:27:54.092 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.213 2024-09-16T05:27:54.093 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.093 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:27:54.093 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.411, 2024-09-16T05:27:54.093 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.548, 2024-09-16T05:27:54.093 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.548 2024-09-16T05:27:54.093 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.093 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.275 2024-09-16T05:27:54.093 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.093 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:27:54.093 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:27:47 2024", 2024-09-16T05:27:54.093 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:27:54.093 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 0, 2024-09-16T05:27:54.093 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 1, 2024-09-16T05:27:54.094 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "front", 2024-09-16T05:27:54.094 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:27:54.094 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.322, 2024-09-16T05:27:54.094 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.313, 2024-09-16T05:27:54.094 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.295 2024-09-16T05:27:54.094 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.094 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:27:54.094 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.250, 2024-09-16T05:27:54.094 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.218, 2024-09-16T05:27:54.094 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.194 2024-09-16T05:27:54.094 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.094 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:27:54.094 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.477, 2024-09-16T05:27:54.095 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.517, 2024-09-16T05:27:54.095 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.517 2024-09-16T05:27:54.095 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.095 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.234 2024-09-16T05:27:54.095 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.095 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:27:54.095 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:27:47 2024", 2024-09-16T05:27:54.095 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:27:54.095 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 1, 2024-09-16T05:27:54.095 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 2, 2024-09-16T05:27:54.095 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "back", 2024-09-16T05:27:54.095 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:27:54.095 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.317, 2024-09-16T05:27:54.095 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.313, 2024-09-16T05:27:54.095 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.307 2024-09-16T05:27:54.096 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.096 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:27:54.096 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.248, 2024-09-16T05:27:54.096 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.214, 2024-09-16T05:27:54.096 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.214 2024-09-16T05:27:54.096 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.096 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:27:54.096 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.423, 2024-09-16T05:27:54.096 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.591, 2024-09-16T05:27:54.096 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.591 2024-09-16T05:27:54.096 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.096 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.302 2024-09-16T05:27:54.096 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.097 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:27:54.097 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:27:47 2024", 2024-09-16T05:27:54.097 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:27:54.097 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 0, 2024-09-16T05:27:54.097 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 1, 2024-09-16T05:27:54.097 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "back", 2024-09-16T05:27:54.097 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:27:54.097 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.300, 2024-09-16T05:27:54.097 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.299, 2024-09-16T05:27:54.097 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.283 2024-09-16T05:27:54.097 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.097 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:27:54.097 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.213, 2024-09-16T05:27:54.097 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.200, 2024-09-16T05:27:54.097 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.179 2024-09-16T05:27:54.098 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.098 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:27:54.098 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.498, 2024-09-16T05:27:54.098 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.498, 2024-09-16T05:27:54.098 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.498 2024-09-16T05:27:54.098 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.098 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.188 2024-09-16T05:27:54.098 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.098 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:27:54.098 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:27:51 2024", 2024-09-16T05:27:54.098 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:27:54.098 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 1, 2024-09-16T05:27:54.098 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 0, 2024-09-16T05:27:54.098 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "front", 2024-09-16T05:27:54.099 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:27:54.099 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.287, 2024-09-16T05:27:54.099 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.291, 2024-09-16T05:27:54.099 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.278 2024-09-16T05:27:54.099 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.099 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:27:54.099 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.237, 2024-09-16T05:27:54.099 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.198, 2024-09-16T05:27:54.099 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.184 2024-09-16T05:27:54.099 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.099 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:27:54.099 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.391, 2024-09-16T05:27:54.099 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.465, 2024-09-16T05:27:54.099 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.482 2024-09-16T05:27:54.099 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.100 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.289 2024-09-16T05:27:54.100 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.100 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:27:54.100 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:27:51 2024", 2024-09-16T05:27:54.100 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:27:54.100 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 1, 2024-09-16T05:27:54.100 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 0, 2024-09-16T05:27:54.100 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "back", 2024-09-16T05:27:54.100 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:27:54.100 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.273, 2024-09-16T05:27:54.100 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.288, 2024-09-16T05:27:54.100 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.273 2024-09-16T05:27:54.100 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.100 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:27:54.101 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.222, 2024-09-16T05:27:54.101 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.179, 2024-09-16T05:27:54.101 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.169 2024-09-16T05:27:54.101 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.101 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:27:54.101 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.348, 2024-09-16T05:27:54.101 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.510, 2024-09-16T05:27:54.101 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.510 2024-09-16T05:27:54.101 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:27:54.101 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.320 2024-09-16T05:27:54.102 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T05:27:54.102 INFO:tasks.workunit.client.0.smithi178.stdout: ] 2024-09-16T05:27:54.102 INFO:tasks.workunit.client.0.smithi178.stdout:} 2024-09-16T05:27:54.103 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T05:27:54.103 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: jq '.entries | length' 2024-09-16T05:27:54.103 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: test 12 = 12 2024-09-16T05:27:54.103 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T05:27:54.103 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: jq .threshold 2024-09-16T05:27:54.103 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: test 0 = 0 2024-09-16T05:27:54.103 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:61: TEST_network_ping_test1: sleep 20 2024-09-16T05:28:14.095 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:62: TEST_network_ping_test1: flush_pg_stats 2024-09-16T05:28:14.096 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-16T05:28:14.096 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-16T05:28:14.475 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-16T05:28:14.475 INFO:tasks.workunit.client.0.smithi178.stderr:1 2024-09-16T05:28:14.475 INFO:tasks.workunit.client.0.smithi178.stderr:2' 2024-09-16T05:28:14.475 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-16T05:28:14.476 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:28:14.476 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T05:28:14.588 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836510 2024-09-16T05:28:14.588 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836510 2024-09-16T05:28:14.588 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510' 2024-09-16T05:28:14.588 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:28:14.588 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T05:28:14.707 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672987 2024-09-16T05:28:14.708 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672987 2024-09-16T05:28:14.708 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-42949672987' 2024-09-16T05:28:14.708 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:28:14.708 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T05:28:14.820 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509460 2024-09-16T05:28:14.821 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509460 2024-09-16T05:28:14.821 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-42949672987 2-64424509460' 2024-09-16T05:28:14.821 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:28:14.821 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836510 2024-09-16T05:28:14.821 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:28:14.821 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-16T05:28:14.822 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836510 2024-09-16T05:28:14.822 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:28:14.823 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836510 2024-09-16T05:28:14.823 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-09-16T05:28:14.823 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.0 seq 21474836510 2024-09-16T05:28:14.824 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:28:15.234 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836510 -lt 21474836510 2024-09-16T05:28:15.234 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:28:15.235 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672987 2024-09-16T05:28:15.235 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:28:15.235 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-09-16T05:28:15.236 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672987 2024-09-16T05:28:15.236 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:28:15.237 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672987 2024-09-16T05:28:15.237 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672987' 2024-09-16T05:28:15.237 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.1 seq 42949672987 2024-09-16T05:28:15.238 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T05:28:15.608 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672985 -lt 42949672987 2024-09-16T05:28:15.608 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-16T05:28:16.609 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T05:28:16.610 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T05:28:17.013 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672987 -lt 42949672987 2024-09-16T05:28:17.013 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:28:17.013 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509460 2024-09-16T05:28:17.013 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:28:17.014 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-09-16T05:28:17.015 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509460 2024-09-16T05:28:17.015 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:28:17.016 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509460 2024-09-16T05:28:17.016 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.2 seq 64424509460 2024-09-16T05:28:17.016 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-09-16T05:28:17.016 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T05:28:17.454 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509460 -lt 64424509460 2024-09-16T05:28:17.455 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: tee td/network-ping/json 2024-09-16T05:28:17.456 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: get_asok_path osd.0 2024-09-16T05:28:17.456 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:28:17.456 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:28:17.456 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:28:17.456 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:28:17.456 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41120 2024-09-16T05:28:17.456 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41120/ceph-osd.0.asok 2024-09-16T05:28:17.456 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: CEPH_ARGS= 2024-09-16T05:28:17.456 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41120/ceph-osd.0.asok dump_osd_network 2024-09-16T05:28:17.533 INFO:tasks.workunit.client.0.smithi178.stdout:{ 2024-09-16T05:28:17.533 INFO:tasks.workunit.client.0.smithi178.stdout: "threshold": 1000, 2024-09-16T05:28:17.533 INFO:tasks.workunit.client.0.smithi178.stdout: "entries": [] 2024-09-16T05:28:17.533 INFO:tasks.workunit.client.0.smithi178.stdout:} 2024-09-16T05:28:17.540 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T05:28:17.541 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: jq '.entries | length' 2024-09-16T05:28:17.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: test 0 = 0 2024-09-16T05:28:17.544 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T05:28:17.544 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: jq .threshold 2024-09-16T05:28:17.546 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: test 1000 = 1000 2024-09-16T05:28:17.547 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: tee td/network-ping/json 2024-09-16T05:28:17.547 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-16T05:28:17.547 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-09-16T05:28:17.548 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-09-16T05:28:17.548 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:28:17.548 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:28:17.548 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41120 2024-09-16T05:28:17.548 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41120/ceph-mgr.x.asok 2024-09-16T05:28:17.548 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: CEPH_ARGS= 2024-09-16T05:28:17.548 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41120/ceph-mgr.x.asok dump_osd_network 2024-09-16T05:28:17.618 INFO:tasks.workunit.client.0.smithi178.stdout:{ 2024-09-16T05:28:17.619 INFO:tasks.workunit.client.0.smithi178.stdout: "threshold": 1000, 2024-09-16T05:28:17.619 INFO:tasks.workunit.client.0.smithi178.stdout: "entries": [] 2024-09-16T05:28:17.619 INFO:tasks.workunit.client.0.smithi178.stdout:} 2024-09-16T05:28:17.627 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T05:28:17.627 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: jq '.entries | length' 2024-09-16T05:28:17.629 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: test 0 = 0 2024-09-16T05:28:17.629 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T05:28:17.630 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: jq .threshold 2024-09-16T05:28:17.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: test 1000 = 1000 2024-09-16T05:28:17.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: tee td/network-ping/json 2024-09-16T05:28:17.633 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: get_asok_path osd.0 2024-09-16T05:28:17.633 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:28:17.634 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:28:17.634 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:28:17.634 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:28:17.634 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41120 2024-09-16T05:28:17.634 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41120/ceph-osd.0.asok 2024-09-16T05:28:17.635 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: CEPH_ARGS= 2024-09-16T05:28:17.635 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41120/ceph-osd.0.asok dump_osd_network 0 2024-09-16T05:28:17.711 INFO:tasks.workunit.client.0.smithi178.stdout:{ 2024-09-16T05:28:17.711 INFO:tasks.workunit.client.0.smithi178.stdout: "threshold": 0, 2024-09-16T05:28:17.711 INFO:tasks.workunit.client.0.smithi178.stdout: "entries": [ 2024-09-16T05:28:17.711 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:28:17.711 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:28:13 2024", 2024-09-16T05:28:17.711 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:28:17.711 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 0, 2024-09-16T05:28:17.711 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 2, 2024-09-16T05:28:17.711 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "front", 2024-09-16T05:28:17.711 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:28:17.712 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.341, 2024-09-16T05:28:17.712 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.349, 2024-09-16T05:28:17.712 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.345 2024-09-16T05:28:17.712 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.712 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:28:17.712 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.455, 2024-09-16T05:28:17.712 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.494, 2024-09-16T05:28:17.712 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.572 2024-09-16T05:28:17.712 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.712 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:28:17.712 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.455, 2024-09-16T05:28:17.712 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.494, 2024-09-16T05:28:17.712 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.572 2024-09-16T05:28:17.713 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.713 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.205 2024-09-16T05:28:17.713 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.713 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:28:17.713 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:28:13 2024", 2024-09-16T05:28:17.713 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:28:17.713 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 0, 2024-09-16T05:28:17.713 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 2, 2024-09-16T05:28:17.713 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "back", 2024-09-16T05:28:17.713 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:28:17.713 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.327, 2024-09-16T05:28:17.713 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.329, 2024-09-16T05:28:17.713 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.328 2024-09-16T05:28:17.714 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.714 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:28:17.714 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.470, 2024-09-16T05:28:17.714 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.470, 2024-09-16T05:28:17.714 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.470 2024-09-16T05:28:17.714 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.714 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:28:17.714 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.470, 2024-09-16T05:28:17.714 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.470, 2024-09-16T05:28:17.714 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.470 2024-09-16T05:28:17.714 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.714 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.216 2024-09-16T05:28:17.714 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.715 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:28:17.715 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:28:17 2024", 2024-09-16T05:28:17.715 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:28:17.715 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 0, 2024-09-16T05:28:17.715 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 1, 2024-09-16T05:28:17.715 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "front", 2024-09-16T05:28:17.715 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:28:17.715 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.226, 2024-09-16T05:28:17.715 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.323, 2024-09-16T05:28:17.715 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.304 2024-09-16T05:28:17.715 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.715 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:28:17.715 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.308, 2024-09-16T05:28:17.716 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.530, 2024-09-16T05:28:17.716 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.530 2024-09-16T05:28:17.716 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.716 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:28:17.716 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.308, 2024-09-16T05:28:17.716 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.530, 2024-09-16T05:28:17.716 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.530 2024-09-16T05:28:17.716 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.716 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.235 2024-09-16T05:28:17.716 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.716 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:28:17.716 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:28:17 2024", 2024-09-16T05:28:17.716 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:28:17.717 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 0, 2024-09-16T05:28:17.717 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 1, 2024-09-16T05:28:17.717 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "back", 2024-09-16T05:28:17.717 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:28:17.717 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.222, 2024-09-16T05:28:17.717 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.314, 2024-09-16T05:28:17.717 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.294 2024-09-16T05:28:17.717 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.717 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:28:17.717 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.267, 2024-09-16T05:28:17.717 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.487, 2024-09-16T05:28:17.718 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.498 2024-09-16T05:28:17.718 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.718 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:28:17.718 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.267, 2024-09-16T05:28:17.718 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.487, 2024-09-16T05:28:17.718 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.498 2024-09-16T05:28:17.718 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.718 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.175 2024-09-16T05:28:17.718 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T05:28:17.718 INFO:tasks.workunit.client.0.smithi178.stdout: ] 2024-09-16T05:28:17.718 INFO:tasks.workunit.client.0.smithi178.stdout:} 2024-09-16T05:28:17.719 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T05:28:17.719 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: jq '.entries | length' 2024-09-16T05:28:17.721 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: test 4 = 4 2024-09-16T05:28:17.722 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T05:28:17.722 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: jq .threshold 2024-09-16T05:28:17.724 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: test 0 = 0 2024-09-16T05:28:17.725 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: tee td/network-ping/json 2024-09-16T05:28:17.725 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-16T05:28:17.725 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-09-16T05:28:17.726 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-09-16T05:28:17.726 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:28:17.726 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:28:17.726 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41120 2024-09-16T05:28:17.726 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41120/ceph-mgr.x.asok 2024-09-16T05:28:17.726 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: CEPH_ARGS= 2024-09-16T05:28:17.726 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41120/ceph-mgr.x.asok dump_osd_network 0 2024-09-16T05:28:17.795 INFO:tasks.workunit.client.0.smithi178.stdout:{ 2024-09-16T05:28:17.795 INFO:tasks.workunit.client.0.smithi178.stdout: "threshold": 0, 2024-09-16T05:28:17.795 INFO:tasks.workunit.client.0.smithi178.stdout: "entries": [ 2024-09-16T05:28:17.795 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:28:17.795 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:28:12 2024", 2024-09-16T05:28:17.796 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:28:17.796 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 2, 2024-09-16T05:28:17.796 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 1, 2024-09-16T05:28:17.796 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "front", 2024-09-16T05:28:17.796 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:28:17.796 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.349, 2024-09-16T05:28:17.796 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.327, 2024-09-16T05:28:17.796 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.336 2024-09-16T05:28:17.796 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.796 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:28:17.796 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.290, 2024-09-16T05:28:17.796 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.218, 2024-09-16T05:28:17.796 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.218 2024-09-16T05:28:17.797 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.797 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:28:17.797 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.408, 2024-09-16T05:28:17.797 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.434, 2024-09-16T05:28:17.797 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.434 2024-09-16T05:28:17.797 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.797 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.259 2024-09-16T05:28:17.797 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.797 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:28:17.797 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:28:13 2024", 2024-09-16T05:28:17.797 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:28:17.797 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 0, 2024-09-16T05:28:17.797 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 2, 2024-09-16T05:28:17.798 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "front", 2024-09-16T05:28:17.798 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:28:17.798 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.341, 2024-09-16T05:28:17.798 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.349, 2024-09-16T05:28:17.798 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.345 2024-09-16T05:28:17.798 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.798 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:28:17.798 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.285, 2024-09-16T05:28:17.798 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.206, 2024-09-16T05:28:17.798 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.206 2024-09-16T05:28:17.798 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.798 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:28:17.798 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.455, 2024-09-16T05:28:17.799 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.494, 2024-09-16T05:28:17.799 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.572 2024-09-16T05:28:17.799 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.799 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.223 2024-09-16T05:28:17.799 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.799 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:28:17.799 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:28:12 2024", 2024-09-16T05:28:17.799 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:28:17.799 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 2, 2024-09-16T05:28:17.799 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 0, 2024-09-16T05:28:17.799 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "front", 2024-09-16T05:28:17.799 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:28:17.799 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.347, 2024-09-16T05:28:17.799 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.314, 2024-09-16T05:28:17.800 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.316 2024-09-16T05:28:17.800 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.800 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:28:17.800 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.282, 2024-09-16T05:28:17.800 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.208, 2024-09-16T05:28:17.800 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.194 2024-09-16T05:28:17.800 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.800 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:28:17.800 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.381, 2024-09-16T05:28:17.800 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.456, 2024-09-16T05:28:17.800 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.456 2024-09-16T05:28:17.800 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.800 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.241 2024-09-16T05:28:17.801 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.801 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:28:17.801 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:28:12 2024", 2024-09-16T05:28:17.801 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:28:17.801 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 0, 2024-09-16T05:28:17.801 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 1, 2024-09-16T05:28:17.801 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "front", 2024-09-16T05:28:17.801 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:28:17.801 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.339, 2024-09-16T05:28:17.801 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.338, 2024-09-16T05:28:17.801 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.306 2024-09-16T05:28:17.801 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.801 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:28:17.802 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.247, 2024-09-16T05:28:17.802 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.201, 2024-09-16T05:28:17.802 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.194 2024-09-16T05:28:17.802 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.802 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:28:17.802 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.491, 2024-09-16T05:28:17.802 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.530, 2024-09-16T05:28:17.802 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.530 2024-09-16T05:28:17.802 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.802 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.170 2024-09-16T05:28:17.802 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.802 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:28:17.802 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:28:12 2024", 2024-09-16T05:28:17.802 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:28:17.803 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 1, 2024-09-16T05:28:17.803 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 2, 2024-09-16T05:28:17.803 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "front", 2024-09-16T05:28:17.803 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:28:17.803 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.330, 2024-09-16T05:28:17.803 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.329, 2024-09-16T05:28:17.803 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.315 2024-09-16T05:28:17.803 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.803 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:28:17.803 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.272, 2024-09-16T05:28:17.803 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.234, 2024-09-16T05:28:17.803 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.213 2024-09-16T05:28:17.803 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.804 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:28:17.804 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.416, 2024-09-16T05:28:17.804 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.462, 2024-09-16T05:28:17.804 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.548 2024-09-16T05:28:17.804 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.804 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.425 2024-09-16T05:28:17.804 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.804 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:28:17.804 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:28:13 2024", 2024-09-16T05:28:17.804 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:28:17.804 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 0, 2024-09-16T05:28:17.804 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 2, 2024-09-16T05:28:17.804 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "back", 2024-09-16T05:28:17.805 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:28:17.805 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.327, 2024-09-16T05:28:17.805 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.329, 2024-09-16T05:28:17.805 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.328 2024-09-16T05:28:17.805 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.805 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:28:17.805 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.273, 2024-09-16T05:28:17.805 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.212, 2024-09-16T05:28:17.805 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.212 2024-09-16T05:28:17.805 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.805 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:28:17.805 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.470, 2024-09-16T05:28:17.805 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.470, 2024-09-16T05:28:17.806 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.470 2024-09-16T05:28:17.806 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.806 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.235 2024-09-16T05:28:17.806 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.806 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:28:17.806 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:28:12 2024", 2024-09-16T05:28:17.806 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:28:17.806 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 2, 2024-09-16T05:28:17.806 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 1, 2024-09-16T05:28:17.806 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "back", 2024-09-16T05:28:17.806 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:28:17.806 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.319, 2024-09-16T05:28:17.806 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.322, 2024-09-16T05:28:17.806 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.327 2024-09-16T05:28:17.807 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.807 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:28:17.807 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.237, 2024-09-16T05:28:17.807 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.208, 2024-09-16T05:28:17.807 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.208 2024-09-16T05:28:17.807 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.807 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:28:17.807 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.369, 2024-09-16T05:28:17.807 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.449, 2024-09-16T05:28:17.807 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.449 2024-09-16T05:28:17.807 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.807 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.206 2024-09-16T05:28:17.807 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.807 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:28:17.808 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:28:12 2024", 2024-09-16T05:28:17.808 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:28:17.808 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 0, 2024-09-16T05:28:17.808 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 1, 2024-09-16T05:28:17.808 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "back", 2024-09-16T05:28:17.808 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:28:17.808 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.310, 2024-09-16T05:28:17.808 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.322, 2024-09-16T05:28:17.808 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.296 2024-09-16T05:28:17.808 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.808 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:28:17.808 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.204, 2024-09-16T05:28:17.808 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.179, 2024-09-16T05:28:17.808 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.179 2024-09-16T05:28:17.809 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.809 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:28:17.809 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.430, 2024-09-16T05:28:17.809 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.487, 2024-09-16T05:28:17.809 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.498 2024-09-16T05:28:17.809 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.809 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.205 2024-09-16T05:28:17.809 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.809 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:28:17.809 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:28:12 2024", 2024-09-16T05:28:17.809 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:28:17.809 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 1, 2024-09-16T05:28:17.812 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 2, 2024-09-16T05:28:17.812 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "back", 2024-09-16T05:28:17.812 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:28:17.812 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.306, 2024-09-16T05:28:17.812 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.317, 2024-09-16T05:28:17.812 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.311 2024-09-16T05:28:17.813 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.813 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:28:17.813 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.251, 2024-09-16T05:28:17.813 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.219, 2024-09-16T05:28:17.813 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.214 2024-09-16T05:28:17.813 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.813 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:28:17.813 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.343, 2024-09-16T05:28:17.813 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.423, 2024-09-16T05:28:17.813 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.591 2024-09-16T05:28:17.813 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.813 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.369 2024-09-16T05:28:17.813 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.813 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:28:17.814 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:28:12 2024", 2024-09-16T05:28:17.814 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:28:17.814 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 2, 2024-09-16T05:28:17.814 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 0, 2024-09-16T05:28:17.814 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "back", 2024-09-16T05:28:17.814 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:28:17.814 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.298, 2024-09-16T05:28:17.814 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.306, 2024-09-16T05:28:17.814 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.312 2024-09-16T05:28:17.814 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.814 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:28:17.814 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.255, 2024-09-16T05:28:17.814 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.221, 2024-09-16T05:28:17.814 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.221 2024-09-16T05:28:17.815 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.815 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:28:17.815 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.348, 2024-09-16T05:28:17.815 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.449, 2024-09-16T05:28:17.815 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.455 2024-09-16T05:28:17.815 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.815 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.223 2024-09-16T05:28:17.815 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.815 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:28:17.815 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:28:11 2024", 2024-09-16T05:28:17.815 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:28:17.815 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 1, 2024-09-16T05:28:17.815 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 0, 2024-09-16T05:28:17.815 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "front", 2024-09-16T05:28:17.816 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:28:17.816 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.294, 2024-09-16T05:28:17.816 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.312, 2024-09-16T05:28:17.816 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.293 2024-09-16T05:28:17.816 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.816 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:28:17.816 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.250, 2024-09-16T05:28:17.816 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.237, 2024-09-16T05:28:17.816 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.184 2024-09-16T05:28:17.816 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.816 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:28:17.816 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.359, 2024-09-16T05:28:17.816 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.437, 2024-09-16T05:28:17.816 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.482 2024-09-16T05:28:17.816 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.817 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.339 2024-09-16T05:28:17.817 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.817 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:28:17.817 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:28:11 2024", 2024-09-16T05:28:17.817 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:28:17.817 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 1, 2024-09-16T05:28:17.817 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 0, 2024-09-16T05:28:17.817 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "back", 2024-09-16T05:28:17.817 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:28:17.817 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.295, 2024-09-16T05:28:17.817 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.303, 2024-09-16T05:28:17.817 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.282 2024-09-16T05:28:17.817 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.817 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:28:17.818 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.232, 2024-09-16T05:28:17.818 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.204, 2024-09-16T05:28:17.818 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.178 2024-09-16T05:28:17.818 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.818 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:28:17.818 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.417, 2024-09-16T05:28:17.818 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.417, 2024-09-16T05:28:17.818 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.510 2024-09-16T05:28:17.818 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:28:17.818 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.385 2024-09-16T05:28:17.818 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T05:28:17.818 INFO:tasks.workunit.client.0.smithi178.stdout: ] 2024-09-16T05:28:17.818 INFO:tasks.workunit.client.0.smithi178.stdout:} 2024-09-16T05:28:17.819 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T05:28:17.820 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: jq '.entries | length' 2024-09-16T05:28:17.820 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: test 12 = 12 2024-09-16T05:28:17.820 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T05:28:17.820 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: jq .threshold 2024-09-16T05:28:17.820 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: test 0 = 0 2024-09-16T05:28:17.820 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:81: TEST_network_ping_test1: sleep 50 2024-09-16T05:29:07.811 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:82: TEST_network_ping_test1: flush_pg_stats 2024-09-16T05:29:07.811 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-16T05:29:07.811 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-16T05:29:08.229 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-16T05:29:08.229 INFO:tasks.workunit.client.0.smithi178.stderr:1 2024-09-16T05:29:08.229 INFO:tasks.workunit.client.0.smithi178.stderr:2' 2024-09-16T05:29:08.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-16T05:29:08.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:29:08.230 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T05:29:08.340 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836523 2024-09-16T05:29:08.341 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836523 2024-09-16T05:29:08.341 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836523' 2024-09-16T05:29:08.341 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:29:08.341 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T05:29:08.454 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673000 2024-09-16T05:29:08.454 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673000 2024-09-16T05:29:08.454 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836523 1-42949673000' 2024-09-16T05:29:08.454 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:29:08.454 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T05:29:08.566 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509473 2024-09-16T05:29:08.567 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509473 2024-09-16T05:29:08.567 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836523 1-42949673000 2-64424509473' 2024-09-16T05:29:08.567 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:29:08.567 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836523 2024-09-16T05:29:08.568 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:29:08.568 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-16T05:29:08.569 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836523 2024-09-16T05:29:08.569 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:29:08.570 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836523 2024-09-16T05:29:08.570 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.0 seq 21474836523 2024-09-16T05:29:08.570 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836523' 2024-09-16T05:29:08.570 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:29:08.941 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836523 -lt 21474836523 2024-09-16T05:29:08.941 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:29:08.942 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673000 2024-09-16T05:29:08.942 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:29:08.942 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-09-16T05:29:08.943 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673000 2024-09-16T05:29:08.943 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:29:08.944 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673000 2024-09-16T05:29:08.944 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673000' 2024-09-16T05:29:08.945 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.1 seq 42949673000 2024-09-16T05:29:08.945 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T05:29:09.316 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673000 -lt 42949673000 2024-09-16T05:29:09.316 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:29:09.317 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509473 2024-09-16T05:29:09.317 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:29:09.317 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-09-16T05:29:09.318 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509473 2024-09-16T05:29:09.318 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:29:09.319 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509473 2024-09-16T05:29:09.319 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509473' 2024-09-16T05:29:09.319 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.2 seq 64424509473 2024-09-16T05:29:09.320 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T05:29:09.690 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509473 -lt 64424509473 2024-09-16T05:29:09.690 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: tee td/network-ping/json 2024-09-16T05:29:09.690 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: get_asok_path osd.0 2024-09-16T05:29:09.690 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:29:09.690 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:29:09.690 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:29:09.690 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:29:09.690 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41120 2024-09-16T05:29:09.690 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41120/ceph-osd.0.asok 2024-09-16T05:29:09.691 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: CEPH_ARGS= 2024-09-16T05:29:09.691 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41120/ceph-osd.0.asok dump_osd_network 2024-09-16T05:29:09.768 INFO:tasks.workunit.client.0.smithi178.stdout:{ 2024-09-16T05:29:09.768 INFO:tasks.workunit.client.0.smithi178.stdout: "threshold": 1000, 2024-09-16T05:29:09.768 INFO:tasks.workunit.client.0.smithi178.stdout: "entries": [] 2024-09-16T05:29:09.768 INFO:tasks.workunit.client.0.smithi178.stdout:} 2024-09-16T05:29:09.777 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T05:29:09.777 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: jq '.entries | length' 2024-09-16T05:29:09.779 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: test 0 = 0 2024-09-16T05:29:09.780 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T05:29:09.780 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: jq .threshold 2024-09-16T05:29:09.782 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: test 1000 = 1000 2024-09-16T05:29:09.783 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: tee td/network-ping/json 2024-09-16T05:29:09.784 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-16T05:29:09.784 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-09-16T05:29:09.784 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-09-16T05:29:09.784 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:29:09.784 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:29:09.784 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41120 2024-09-16T05:29:09.784 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41120/ceph-mgr.x.asok 2024-09-16T05:29:09.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: CEPH_ARGS= 2024-09-16T05:29:09.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41120/ceph-mgr.x.asok dump_osd_network 2024-09-16T05:29:09.854 INFO:tasks.workunit.client.0.smithi178.stdout:{ 2024-09-16T05:29:09.854 INFO:tasks.workunit.client.0.smithi178.stdout: "threshold": 1000, 2024-09-16T05:29:09.855 INFO:tasks.workunit.client.0.smithi178.stdout: "entries": [] 2024-09-16T05:29:09.855 INFO:tasks.workunit.client.0.smithi178.stdout:} 2024-09-16T05:29:09.862 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T05:29:09.862 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: jq '.entries | length' 2024-09-16T05:29:09.864 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: test 0 = 0 2024-09-16T05:29:09.865 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T05:29:09.865 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: jq .threshold 2024-09-16T05:29:09.867 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: test 1000 = 1000 2024-09-16T05:29:09.868 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: tee td/network-ping/json 2024-09-16T05:29:09.868 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: get_asok_path osd.0 2024-09-16T05:29:09.868 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:29:09.869 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:29:09.869 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:29:09.869 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:29:09.869 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41120 2024-09-16T05:29:09.869 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41120/ceph-osd.0.asok 2024-09-16T05:29:09.869 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: CEPH_ARGS= 2024-09-16T05:29:09.869 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41120/ceph-osd.0.asok dump_osd_network 0 2024-09-16T05:29:09.946 INFO:tasks.workunit.client.0.smithi178.stdout:{ 2024-09-16T05:29:09.946 INFO:tasks.workunit.client.0.smithi178.stdout: "threshold": 0, 2024-09-16T05:29:09.946 INFO:tasks.workunit.client.0.smithi178.stdout: "entries": [ 2024-09-16T05:29:09.946 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:29:09.946 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:29:08 2024", 2024-09-16T05:29:09.946 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:29:09.947 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 0, 2024-09-16T05:29:09.947 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 2, 2024-09-16T05:29:09.947 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "front", 2024-09-16T05:29:09.947 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:29:09.947 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.319, 2024-09-16T05:29:09.947 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.339, 2024-09-16T05:29:09.947 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.331 2024-09-16T05:29:09.947 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:09.947 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:29:09.947 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.448, 2024-09-16T05:29:09.947 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.487, 2024-09-16T05:29:09.947 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.494 2024-09-16T05:29:09.947 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:09.947 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:29:09.948 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.448, 2024-09-16T05:29:09.948 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.487, 2024-09-16T05:29:09.948 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.494 2024-09-16T05:29:09.948 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:09.948 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.236 2024-09-16T05:29:09.948 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:09.948 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:29:09.948 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:29:08 2024", 2024-09-16T05:29:09.948 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:29:09.948 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 0, 2024-09-16T05:29:09.948 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 2, 2024-09-16T05:29:09.948 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "back", 2024-09-16T05:29:09.948 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:29:09.949 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.287, 2024-09-16T05:29:09.949 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.331, 2024-09-16T05:29:09.949 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.323 2024-09-16T05:29:09.949 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:09.949 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:29:09.949 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.385, 2024-09-16T05:29:09.949 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.468, 2024-09-16T05:29:09.949 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.470 2024-09-16T05:29:09.949 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:09.949 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:29:09.949 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.385, 2024-09-16T05:29:09.949 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.468, 2024-09-16T05:29:09.949 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.470 2024-09-16T05:29:09.949 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:09.950 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.226 2024-09-16T05:29:09.950 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:09.950 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:29:09.950 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:29:07 2024", 2024-09-16T05:29:09.950 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:29:09.950 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 0, 2024-09-16T05:29:09.950 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 1, 2024-09-16T05:29:09.950 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "front", 2024-09-16T05:29:09.950 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:29:09.950 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.284, 2024-09-16T05:29:09.950 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.319, 2024-09-16T05:29:09.950 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.316 2024-09-16T05:29:09.950 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:09.950 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:29:09.951 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.406, 2024-09-16T05:29:09.951 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.453, 2024-09-16T05:29:09.951 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.530 2024-09-16T05:29:09.951 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:09.951 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:29:09.951 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.406, 2024-09-16T05:29:09.952 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.453, 2024-09-16T05:29:09.952 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.530 2024-09-16T05:29:09.952 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:09.952 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.204 2024-09-16T05:29:09.952 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:09.952 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:29:09.952 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:29:07 2024", 2024-09-16T05:29:09.952 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:29:09.952 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 0, 2024-09-16T05:29:09.952 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 1, 2024-09-16T05:29:09.952 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "back", 2024-09-16T05:29:09.952 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:29:09.952 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.284, 2024-09-16T05:29:09.952 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.313, 2024-09-16T05:29:09.953 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.304 2024-09-16T05:29:09.953 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:09.953 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:29:09.953 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.353, 2024-09-16T05:29:09.953 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.425, 2024-09-16T05:29:09.953 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.487 2024-09-16T05:29:09.953 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:09.953 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:29:09.953 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.353, 2024-09-16T05:29:09.953 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.425, 2024-09-16T05:29:09.953 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.487 2024-09-16T05:29:09.953 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:09.953 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.186 2024-09-16T05:29:09.953 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T05:29:09.953 INFO:tasks.workunit.client.0.smithi178.stdout: ] 2024-09-16T05:29:09.954 INFO:tasks.workunit.client.0.smithi178.stdout:} 2024-09-16T05:29:09.954 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T05:29:09.955 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: jq '.entries | length' 2024-09-16T05:29:09.956 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: test 4 = 4 2024-09-16T05:29:09.956 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T05:29:09.957 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: jq .threshold 2024-09-16T05:29:09.959 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: test 0 = 0 2024-09-16T05:29:09.959 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: tee td/network-ping/json 2024-09-16T05:29:09.960 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-16T05:29:09.960 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-09-16T05:29:09.960 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-09-16T05:29:09.961 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:29:09.961 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:29:09.961 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41120 2024-09-16T05:29:09.961 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41120/ceph-mgr.x.asok 2024-09-16T05:29:09.961 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: CEPH_ARGS= 2024-09-16T05:29:09.961 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41120/ceph-mgr.x.asok dump_osd_network 0 2024-09-16T05:29:10.031 INFO:tasks.workunit.client.0.smithi178.stdout:{ 2024-09-16T05:29:10.031 INFO:tasks.workunit.client.0.smithi178.stdout: "threshold": 0, 2024-09-16T05:29:10.031 INFO:tasks.workunit.client.0.smithi178.stdout: "entries": [ 2024-09-16T05:29:10.031 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:29:10.031 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:29:03 2024", 2024-09-16T05:29:10.031 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:29:10.032 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 0, 2024-09-16T05:29:10.032 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 2, 2024-09-16T05:29:10.032 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "front", 2024-09-16T05:29:10.032 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:29:10.032 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.381, 2024-09-16T05:29:10.032 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.347, 2024-09-16T05:29:10.032 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.329 2024-09-16T05:29:10.032 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.032 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:29:10.032 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.286, 2024-09-16T05:29:10.032 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.243, 2024-09-16T05:29:10.032 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.205 2024-09-16T05:29:10.033 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.033 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:29:10.033 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.487, 2024-09-16T05:29:10.033 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.487, 2024-09-16T05:29:10.033 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.494 2024-09-16T05:29:10.033 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.033 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.285 2024-09-16T05:29:10.033 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.033 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:29:10.033 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:29:03 2024", 2024-09-16T05:29:10.033 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:29:10.033 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 0, 2024-09-16T05:29:10.033 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 2, 2024-09-16T05:29:10.034 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "back", 2024-09-16T05:29:10.034 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:29:10.034 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.359, 2024-09-16T05:29:10.034 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.344, 2024-09-16T05:29:10.034 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.322 2024-09-16T05:29:10.034 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.034 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:29:10.034 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.260, 2024-09-16T05:29:10.034 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.257, 2024-09-16T05:29:10.034 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.196 2024-09-16T05:29:10.034 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.034 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:29:10.034 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.468, 2024-09-16T05:29:10.034 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.468, 2024-09-16T05:29:10.035 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.470 2024-09-16T05:29:10.035 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.035 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.267 2024-09-16T05:29:10.035 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.035 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:29:10.035 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:29:07 2024", 2024-09-16T05:29:10.035 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:29:10.035 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 2, 2024-09-16T05:29:10.035 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 1, 2024-09-16T05:29:10.035 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "front", 2024-09-16T05:29:10.035 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:29:10.035 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.347, 2024-09-16T05:29:10.035 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.337, 2024-09-16T05:29:10.035 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.335 2024-09-16T05:29:10.036 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.036 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:29:10.036 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.289, 2024-09-16T05:29:10.036 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.233, 2024-09-16T05:29:10.037 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.218 2024-09-16T05:29:10.037 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.037 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:29:10.037 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.435, 2024-09-16T05:29:10.037 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.455, 2024-09-16T05:29:10.037 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.455 2024-09-16T05:29:10.037 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.037 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.192 2024-09-16T05:29:10.037 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.037 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:29:10.037 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:29:07 2024", 2024-09-16T05:29:10.037 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:29:10.037 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 2, 2024-09-16T05:29:10.038 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 1, 2024-09-16T05:29:10.038 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "back", 2024-09-16T05:29:10.038 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:29:10.038 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.314, 2024-09-16T05:29:10.038 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.321, 2024-09-16T05:29:10.038 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.325 2024-09-16T05:29:10.038 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.038 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:29:10.038 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.267, 2024-09-16T05:29:10.039 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.245, 2024-09-16T05:29:10.039 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.206 2024-09-16T05:29:10.039 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.039 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:29:10.039 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.390, 2024-09-16T05:29:10.039 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.484, 2024-09-16T05:29:10.039 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.484 2024-09-16T05:29:10.039 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.039 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.229 2024-09-16T05:29:10.040 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.040 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:29:10.040 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:29:07 2024", 2024-09-16T05:29:10.040 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:29:10.040 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 1, 2024-09-16T05:29:10.040 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 2, 2024-09-16T05:29:10.040 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "front", 2024-09-16T05:29:10.040 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:29:10.040 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.277, 2024-09-16T05:29:10.040 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.317, 2024-09-16T05:29:10.040 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.325 2024-09-16T05:29:10.040 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.041 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:29:10.041 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.226, 2024-09-16T05:29:10.041 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.226, 2024-09-16T05:29:10.041 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.207 2024-09-16T05:29:10.041 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.041 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:29:10.041 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.300, 2024-09-16T05:29:10.041 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.464, 2024-09-16T05:29:10.041 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.497 2024-09-16T05:29:10.042 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.042 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.279 2024-09-16T05:29:10.042 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.042 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:29:10.042 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:29:07 2024", 2024-09-16T05:29:10.042 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:29:10.042 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 2, 2024-09-16T05:29:10.042 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 0, 2024-09-16T05:29:10.042 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "front", 2024-09-16T05:29:10.043 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:29:10.043 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.320, 2024-09-16T05:29:10.043 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.319, 2024-09-16T05:29:10.043 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.318 2024-09-16T05:29:10.043 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.043 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:29:10.043 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.268, 2024-09-16T05:29:10.043 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.235, 2024-09-16T05:29:10.043 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.214 2024-09-16T05:29:10.043 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.044 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:29:10.044 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.417, 2024-09-16T05:29:10.044 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.443, 2024-09-16T05:29:10.044 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.456 2024-09-16T05:29:10.044 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.044 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.209 2024-09-16T05:29:10.044 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.044 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:29:10.044 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:29:07 2024", 2024-09-16T05:29:10.044 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:29:10.044 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 2, 2024-09-16T05:29:10.044 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 0, 2024-09-16T05:29:10.045 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "back", 2024-09-16T05:29:10.045 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:29:10.045 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.293, 2024-09-16T05:29:10.045 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.319, 2024-09-16T05:29:10.045 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.312 2024-09-16T05:29:10.045 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.045 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:29:10.045 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.231, 2024-09-16T05:29:10.045 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.231, 2024-09-16T05:29:10.045 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.223 2024-09-16T05:29:10.045 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.045 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:29:10.045 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.401, 2024-09-16T05:29:10.046 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.464, 2024-09-16T05:29:10.046 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.502 2024-09-16T05:29:10.046 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.046 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.245 2024-09-16T05:29:10.046 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.046 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:29:10.046 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:29:07 2024", 2024-09-16T05:29:10.046 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:29:10.046 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 0, 2024-09-16T05:29:10.046 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 1, 2024-09-16T05:29:10.046 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "front", 2024-09-16T05:29:10.046 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:29:10.046 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.284, 2024-09-16T05:29:10.047 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.319, 2024-09-16T05:29:10.047 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.316 2024-09-16T05:29:10.047 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.047 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:29:10.047 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.241, 2024-09-16T05:29:10.047 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.228, 2024-09-16T05:29:10.047 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.170 2024-09-16T05:29:10.047 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.047 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:29:10.047 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.406, 2024-09-16T05:29:10.047 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.453, 2024-09-16T05:29:10.048 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.530 2024-09-16T05:29:10.048 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.048 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.248 2024-09-16T05:29:10.048 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.048 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:29:10.048 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:29:07 2024", 2024-09-16T05:29:10.048 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:29:10.048 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 0, 2024-09-16T05:29:10.048 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 1, 2024-09-16T05:29:10.049 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "back", 2024-09-16T05:29:10.049 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:29:10.049 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.284, 2024-09-16T05:29:10.049 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.313, 2024-09-16T05:29:10.049 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.304 2024-09-16T05:29:10.049 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.049 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:29:10.049 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.234, 2024-09-16T05:29:10.049 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.226, 2024-09-16T05:29:10.049 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.175 2024-09-16T05:29:10.049 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.049 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:29:10.050 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.353, 2024-09-16T05:29:10.050 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.425, 2024-09-16T05:29:10.050 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.487 2024-09-16T05:29:10.050 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.050 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.234 2024-09-16T05:29:10.050 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.050 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:29:10.050 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:29:07 2024", 2024-09-16T05:29:10.050 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:29:10.050 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 1, 2024-09-16T05:29:10.050 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 2, 2024-09-16T05:29:10.050 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "back", 2024-09-16T05:29:10.051 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:29:10.051 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.264, 2024-09-16T05:29:10.051 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.307, 2024-09-16T05:29:10.051 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.312 2024-09-16T05:29:10.051 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.051 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:29:10.051 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.237, 2024-09-16T05:29:10.051 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.237, 2024-09-16T05:29:10.051 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.205 2024-09-16T05:29:10.052 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.052 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:29:10.052 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.305, 2024-09-16T05:29:10.052 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.448, 2024-09-16T05:29:10.052 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.480 2024-09-16T05:29:10.052 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.052 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.266 2024-09-16T05:29:10.052 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.052 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:29:10.053 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:29:06 2024", 2024-09-16T05:29:10.053 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:29:10.053 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 1, 2024-09-16T05:29:10.053 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 0, 2024-09-16T05:29:10.053 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "front", 2024-09-16T05:29:10.053 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:29:10.053 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.272, 2024-09-16T05:29:10.053 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.306, 2024-09-16T05:29:10.053 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.308 2024-09-16T05:29:10.053 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.053 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:29:10.053 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.209, 2024-09-16T05:29:10.053 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.209, 2024-09-16T05:29:10.054 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.209 2024-09-16T05:29:10.054 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.054 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:29:10.054 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.315, 2024-09-16T05:29:10.054 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.425, 2024-09-16T05:29:10.054 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.442 2024-09-16T05:29:10.054 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.054 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.247 2024-09-16T05:29:10.054 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.054 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T05:29:10.054 INFO:tasks.workunit.client.0.smithi178.stdout: "last update": "Mon Sep 16 05:29:06 2024", 2024-09-16T05:29:10.054 INFO:tasks.workunit.client.0.smithi178.stdout: "stale": false, 2024-09-16T05:29:10.055 INFO:tasks.workunit.client.0.smithi178.stdout: "from osd": 1, 2024-09-16T05:29:10.055 INFO:tasks.workunit.client.0.smithi178.stdout: "to osd": 0, 2024-09-16T05:29:10.055 INFO:tasks.workunit.client.0.smithi178.stdout: "interface": "back", 2024-09-16T05:29:10.055 INFO:tasks.workunit.client.0.smithi178.stdout: "average": { 2024-09-16T05:29:10.055 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.267, 2024-09-16T05:29:10.055 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.300, 2024-09-16T05:29:10.055 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.300 2024-09-16T05:29:10.055 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.055 INFO:tasks.workunit.client.0.smithi178.stdout: "min": { 2024-09-16T05:29:10.055 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.223, 2024-09-16T05:29:10.055 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.204, 2024-09-16T05:29:10.055 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.203 2024-09-16T05:29:10.056 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.056 INFO:tasks.workunit.client.0.smithi178.stdout: "max": { 2024-09-16T05:29:10.056 INFO:tasks.workunit.client.0.smithi178.stdout: "1min": 0.344, 2024-09-16T05:29:10.056 INFO:tasks.workunit.client.0.smithi178.stdout: "5min": 0.499, 2024-09-16T05:29:10.056 INFO:tasks.workunit.client.0.smithi178.stdout: "15min": 0.499 2024-09-16T05:29:10.056 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T05:29:10.056 INFO:tasks.workunit.client.0.smithi178.stdout: "last": 0.219 2024-09-16T05:29:10.057 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T05:29:10.057 INFO:tasks.workunit.client.0.smithi178.stdout: ] 2024-09-16T05:29:10.057 INFO:tasks.workunit.client.0.smithi178.stdout:} 2024-09-16T05:29:10.060 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T05:29:10.060 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: jq '.entries | length' 2024-09-16T05:29:10.061 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: test 12 = 12 2024-09-16T05:29:10.061 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T05:29:10.061 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: jq .threshold 2024-09-16T05:29:10.061 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: test 0 = 0 2024-09-16T05:29:10.061 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: tee td/network-ping/json 2024-09-16T05:29:10.061 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-16T05:29:10.061 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-09-16T05:29:10.061 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-09-16T05:29:10.062 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:29:10.062 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:29:10.062 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41120 2024-09-16T05:29:10.062 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41120/ceph-mgr.x.asok 2024-09-16T05:29:10.062 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: CEPH_ARGS= 2024-09-16T05:29:10.062 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41120/ceph-mgr.x.asok dump_osd_network 99 2024-09-16T05:29:10.116 INFO:tasks.workunit.client.0.smithi178.stdout:{ 2024-09-16T05:29:10.116 INFO:tasks.workunit.client.0.smithi178.stdout: "threshold": 99, 2024-09-16T05:29:10.116 INFO:tasks.workunit.client.0.smithi178.stdout: "entries": [] 2024-09-16T05:29:10.116 INFO:tasks.workunit.client.0.smithi178.stdout:} 2024-09-16T05:29:10.124 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T05:29:10.124 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: jq .threshold 2024-09-16T05:29:10.127 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: test 99 = 99 2024-09-16T05:29:10.127 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: tee td/network-ping/json 2024-09-16T05:29:10.127 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: get_asok_path osd.0 2024-09-16T05:29:10.127 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:29:10.127 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:29:10.128 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:29:10.128 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:29:10.128 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41120 2024-09-16T05:29:10.128 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41120/ceph-osd.0.asok 2024-09-16T05:29:10.128 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: CEPH_ARGS= 2024-09-16T05:29:10.128 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.41120/ceph-osd.0.asok dump_osd_network 98 2024-09-16T05:29:10.205 INFO:tasks.workunit.client.0.smithi178.stdout:{ 2024-09-16T05:29:10.205 INFO:tasks.workunit.client.0.smithi178.stdout: "threshold": 98, 2024-09-16T05:29:10.205 INFO:tasks.workunit.client.0.smithi178.stdout: "entries": [] 2024-09-16T05:29:10.205 INFO:tasks.workunit.client.0.smithi178.stdout:} 2024-09-16T05:29:10.212 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T05:29:10.212 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: jq .threshold 2024-09-16T05:29:10.214 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: test 98 = 98 2024-09-16T05:29:10.215 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:105: TEST_network_ping_test1: rm -f td/network-ping/json 2024-09-16T05:29:10.215 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:20: run: teardown td/network-ping 2024-09-16T05:29:10.216 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-09-16T05:29:10.216 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:29:10.216 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-09-16T05:29:10.216 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:29:10.217 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:29:10.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:29:10.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:29:10.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:29:10.324 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:29:10.324 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:29:10.325 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:29:10.325 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:29:10.326 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:29:10.327 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:29:10.327 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:29:10.328 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:29:10.328 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:29:10.328 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:29:10.328 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:29:10.328 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:29:10.329 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:29:10.329 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-09-16T05:29:10.350 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:29:10.350 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:29:10.350 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41120 2024-09-16T05:29:10.350 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.41120 2024-09-16T05:29:10.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:29:10.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:29:10.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:17: run: for func in $funcs 2024-09-16T05:29:10.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:18: run: setup td/network-ping 2024-09-16T05:29:10.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/network-ping 2024-09-16T05:29:10.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/network-ping 2024-09-16T05:29:10.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-09-16T05:29:10.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:29:10.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-09-16T05:29:10.353 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:29:10.353 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:29:10.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:29:10.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:29:10.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:29:10.354 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:29:10.354 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:29:10.354 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:29:10.354 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:29:10.355 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:29:10.355 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:29:10.355 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:29:10.356 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:29:10.356 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:29:10.357 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:29:10.357 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:29:10.357 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:29:10.358 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:29:10.358 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-09-16T05:29:10.360 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:29:10.361 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:29:10.361 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41120 2024-09-16T05:29:10.361 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.41120 2024-09-16T05:29:10.361 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:29:10.361 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:29:10.361 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/network-ping 2024-09-16T05:29:10.362 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:29:10.362 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:29:10.362 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41120 2024-09-16T05:29:10.362 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.41120 2024-09-16T05:29:10.365 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:29:10.365 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:29:10.366 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:29:10.366 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/network-ping 1' TERM HUP INT 2024-09-16T05:29:10.366 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:19: run: TEST_network_ping_test2 td/network-ping 2024-09-16T05:29:10.366 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:111: TEST_network_ping_test2: local dir=td/network-ping 2024-09-16T05:29:10.366 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:113: TEST_network_ping_test2: export CEPH_ARGS 2024-09-16T05:29:10.366 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:114: TEST_network_ping_test2: export 'EXTRA_OPTS= --mon_warn_on_slow_ping_time=0.001' 2024-09-16T05:29:10.366 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:114: TEST_network_ping_test2: EXTRA_OPTS=' --mon_warn_on_slow_ping_time=0.001' 2024-09-16T05:29:10.366 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:115: TEST_network_ping_test2: run_mon td/network-ping a 2024-09-16T05:29:10.367 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/network-ping 2024-09-16T05:29:10.367 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:29:10.367 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:29:10.367 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:29:10.367 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/network-ping/a 2024-09-16T05:29:10.367 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/network-ping/a --run-dir=td/network-ping 2024-09-16T05:29:10.742 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:29:10.742 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:29:10.743 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:29:10.743 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:29:10.743 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:29:10.743 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41120 2024-09-16T05:29:10.743 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.41120/$cluster-$name.asok' 2024-09-16T05:29:10.743 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --mon_warn_on_slow_ping_time=0.001 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/network-ping/a '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.41120/$cluster-$name.asok' --mon-cluster-log-file=td/network-ping/log --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T05:29:10.925 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:29:10.928 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:29:10.928 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:29:10.928 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:29:10.928 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:29:10.928 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:29:10.928 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:29:10.928 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:29:10.928 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:29:10.928 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:29:10.929 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:29:10.929 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41120 2024-09-16T05:29:10.929 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41120/ceph-mon.a.asok 2024-09-16T05:29:10.929 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:29:10.929 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.41120/ceph-mon.a.asok config get fsid 2024-09-16T05:29:11.009 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:29:11.009 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:29:11.009 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:29:11.010 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:29:11.010 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:29:11.010 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:29:11.010 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:29:11.010 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:29:11.010 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:29:11.010 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:29:11.010 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41120 2024-09-16T05:29:11.010 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.41120/ceph-mon.a.asok 2024-09-16T05:29:11.010 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:29:11.010 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.41120/ceph-mon.a.asok config get mon_host 2024-09-16T05:29:11.089 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:116: TEST_network_ping_test2: run_mgr td/network-ping x 2024-09-16T05:29:11.089 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/network-ping 2024-09-16T05:29:11.089 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T05:29:11.089 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T05:29:11.089 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T05:29:11.089 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/network-ping/x 2024-09-16T05:29:11.090 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T05:29:11.964 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T05:29:11.964 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:29:11.965 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:29:11.965 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:29:11.965 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:29:11.965 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41120 2024-09-16T05:29:11.965 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.41120/$cluster-$name.asok' 2024-09-16T05:29:11.965 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:29:11.965 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --mon_warn_on_slow_ping_time=0.001 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/network-ping/x '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.41120/$cluster-$name.asok' --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:29:11.997 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:117: TEST_network_ping_test2: run_osd td/network-ping 0 2024-09-16T05:29:11.997 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-09-16T05:29:11.997 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:29:11.998 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T05:29:11.998 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:29:11.998 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/0 2024-09-16T05:29:11.998 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ad08e9ec-e980-42b8-a051-49ca1d64766e --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-09-16T05:29:11.998 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:29:11.998 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:29:11.998 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:29:11.998 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/0' 2024-09-16T05:29:11.998 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/0/journal' 2024-09-16T05:29:11.998 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:29:11.998 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2024-09-16T05:29:11.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-09-16T05:29:11.999 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:29:11.999 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:29:11.999 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:29:11.999 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:29:11.999 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:29:11.999 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41120 2024-09-16T05:29:11.999 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.41120/$cluster-$name.asok' 2024-09-16T05:29:11.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.41120/$cluster-$name.asok' 2024-09-16T05:29:11.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:29:11.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:29:11.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:29:11.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-09-16T05:29:11.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-09-16T05:29:12.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:29:12.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:29:12.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:29:12.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:29:12.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:29:12.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:29:12.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/0 2024-09-16T05:29:12.000 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:29:12.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3cc28bc3-ef2b-4ce0-b68e-9a50baf58716 2024-09-16T05:29:12.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3cc28bc3-ef2b-4ce0-b68e-9a50baf58716' 2024-09-16T05:29:12.000 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 3cc28bc3-ef2b-4ce0-b68e-9a50baf58716 2024-09-16T05:29:12.001 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:29:12.016 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAowudmudztABAArdRYScO6CR89mhNvKXip3w== 2024-09-16T05:29:12.016 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAowudmudztABAArdRYScO6CR89mhNvKXip3w=="}' 2024-09-16T05:29:12.016 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3cc28bc3-ef2b-4ce0-b68e-9a50baf58716 -i td/network-ping/0/new.json 2024-09-16T05:29:12.284 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:29:12.293 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/0/new.json 2024-09-16T05:29:12.294 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=ad08e9ec-e980-42b8-a051-49ca1d64766e --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41120/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAowudmudztABAArdRYScO6CR89mhNvKXip3w== --osd-uuid 3cc28bc3-ef2b-4ce0-b68e-9a50baf58716 2024-09-16T05:29:12.325 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:29:12.324+0000 7f58bb683640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:29:12.353 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:29:12.352+0000 7f58bb683640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:29:12.353 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:29:12.353+0000 7f58bb683640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:29:12.354 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:29:12.353+0000 7f58bb683640 -1 bdev(0x564d082e1c00 td/network-ping/0/block) open stat got: (1) Operation not permitted 2024-09-16T05:29:12.354 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:29:12.353+0000 7f58bb683640 -1 bluestore(td/network-ping/0) _read_fsid unparsable uuid 2024-09-16T05:29:17.825 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/0/keyring 2024-09-16T05:29:17.826 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:29:17.826 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T05:29:17.827 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T05:29:17.827 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:29:18.345 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:29:18.346 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T05:29:18.346 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=ad08e9ec-e980-42b8-a051-49ca1d64766e --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41120/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:29:18.346 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:29:18.346 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:29:18.346 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:29:18.374 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:29:18.373+0000 7fd5a242e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:29:18.396 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:29:18.396+0000 7fd5a242e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:29:18.397 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:29:18.397+0000 7fd5a242e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:29:18.751 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T05:29:18.751 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:29:18.751 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:29:18.751 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:29:18.752 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:29:18.753 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:29:18.753 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:29:18.753 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:29:18.753 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:29:18.753 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:29:19.160 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:29:19.487 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:29:19.486+0000 7fd5a242e640 -1 Falling back to public interface 2024-09-16T05:29:20.162 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:29:20.162 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:29:20.162 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:29:20.163 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:29:20.163 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:29:20.163 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:29:20.578 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:29:21.579 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:29:21.579 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:29:21.579 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:29:21.580 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:29:21.580 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:29:21.580 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:29:21.953 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:29:22.307 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:29:22.307+0000 7fd5a242e640 -1 osd.0 0 log_to_monitors true 2024-09-16T05:29:22.954 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:29:22.955 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:29:22.955 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:29:22.955 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:29:22.956 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:29:22.956 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:29:23.409 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:29:24.411 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:29:24.411 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:29:24.411 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:29:24.412 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:29:24.412 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:29:24.412 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:29:24.841 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:29:25.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:29:25.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:29:25.842 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:29:25.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:29:25.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:29:25.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:29:26.225 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:29:27.227 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:29:27.227 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:29:27.227 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:29:27.228 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:29:27.228 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:29:27.228 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:29:27.606 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:29:28.607 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:29:28.607 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:29:28.608 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:29:28.608 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:29:28.608 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:29:28.608 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:29:29.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:29:30.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:29:30.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:29:30.044 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:29:30.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:29:30.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:29:30.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:29:30.430 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:29:31.431 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:29:31.431 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:29:31.431 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:29:31.432 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:29:31.432 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:29:31.432 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:29:31.821 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:29:32.823 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:29:32.823 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:29:32.823 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:29:32.824 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:29:32.824 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:29:32.824 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:29:33.206 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:29:34.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:29:34.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:29:34.208 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:29:34.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:29:34.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:29:34.209 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:29:34.802 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:29:35.804 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:29:35.804 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:29:35.805 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:29:35.805 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:29:35.805 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:29:35.805 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:29:36.753 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:29:37.754 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:29:37.754 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:29:37.754 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:29:37.755 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:29:37.755 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:29:37.755 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:29:38.177 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:29:39.178 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:29:39.178 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:29:39.178 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:29:39.179 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:29:39.179 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:29:39.179 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:29:39.929 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:29:40.930 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:29:40.930 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:29:40.930 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:29:40.931 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:29:40.931 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:29:40.931 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:29:41.619 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:29:42.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:29:42.621 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:29:42.622 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:29:42.622 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:29:42.622 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:29:42.622 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:29:43.732 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:29:44.734 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:29:44.734 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:29:44.734 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:29:44.735 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:29:44.735 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:29:44.735 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:29:45.170 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:29:46.171 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:29:46.172 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:29:46.172 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T05:29:46.172 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T05:29:46.172 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:29:46.173 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:29:46.568 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/143057541,v1:127.0.0.1:6803/143057541] [v2:127.0.0.1:6804/143057541,v1:127.0.0.1:6805/143057541] exists,up 3cc28bc3-ef2b-4ce0-b68e-9a50baf58716 2024-09-16T05:29:46.573 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:29:46.573 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:29:46.573 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:29:46.573 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:118: TEST_network_ping_test2: run_osd td/network-ping 1 2024-09-16T05:29:46.573 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-09-16T05:29:46.573 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:29:46.574 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T05:29:46.574 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:29:46.574 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/1 2024-09-16T05:29:46.574 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ad08e9ec-e980-42b8-a051-49ca1d64766e --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-09-16T05:29:46.574 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:29:46.574 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:29:46.574 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:29:46.574 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/1' 2024-09-16T05:29:46.574 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/1/journal' 2024-09-16T05:29:46.574 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:29:46.574 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2024-09-16T05:29:46.575 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-09-16T05:29:46.575 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:29:46.575 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:29:46.575 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:29:46.575 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:29:46.575 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:29:46.575 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41120 2024-09-16T05:29:46.575 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.41120/$cluster-$name.asok' 2024-09-16T05:29:46.575 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.41120/$cluster-$name.asok' 2024-09-16T05:29:46.575 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:29:46.575 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:29:46.575 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:29:46.576 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-09-16T05:29:46.576 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-09-16T05:29:46.576 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:29:46.576 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:29:46.576 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:29:46.576 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:29:46.576 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:29:46.576 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:29:46.576 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/1 2024-09-16T05:29:46.576 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:29:46.576 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4a80cc51-7246-4835-ba5c-16cf5d0ecad7 2024-09-16T05:29:46.577 INFO:tasks.workunit.client.0.smithi178.stdout:add osd1 4a80cc51-7246-4835-ba5c-16cf5d0ecad7 2024-09-16T05:29:46.577 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 4a80cc51-7246-4835-ba5c-16cf5d0ecad7' 2024-09-16T05:29:46.577 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:29:46.591 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBKwudmMnguIxAAF+28D9+rYkirgIyqBEBbYQ== 2024-09-16T05:29:46.591 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBKwudmMnguIxAAF+28D9+rYkirgIyqBEBbYQ=="}' 2024-09-16T05:29:46.591 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4a80cc51-7246-4835-ba5c-16cf5d0ecad7 -i td/network-ping/1/new.json 2024-09-16T05:29:47.027 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:29:47.038 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/1/new.json 2024-09-16T05:29:47.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=ad08e9ec-e980-42b8-a051-49ca1d64766e --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41120/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBKwudmMnguIxAAF+28D9+rYkirgIyqBEBbYQ== --osd-uuid 4a80cc51-7246-4835-ba5c-16cf5d0ecad7 2024-09-16T05:29:47.070 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:29:47.069+0000 7f065e813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:29:47.072 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:29:47.071+0000 7f065e813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:29:47.073 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:29:47.072+0000 7f065e813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:29:47.073 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:29:47.072+0000 7f065e813640 -1 bdev(0x5578a74bbc00 td/network-ping/1/block) open stat got: (1) Operation not permitted 2024-09-16T05:29:47.073 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:29:47.073+0000 7f065e813640 -1 bluestore(td/network-ping/1) _read_fsid unparsable uuid 2024-09-16T05:29:52.836 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/1/keyring 2024-09-16T05:29:52.836 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:29:52.837 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd1 key to auth repository 2024-09-16T05:29:52.838 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T05:29:52.838 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:29:53.347 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T05:29:53.348 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.1 2024-09-16T05:29:53.348 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=ad08e9ec-e980-42b8-a051-49ca1d64766e --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41120/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:29:53.348 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:29:53.348 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:29:53.349 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:29:53.376 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:29:53.376+0000 7f5180c14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:29:53.407 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:29:53.406+0000 7f5180c14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:29:53.408 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:29:53.407+0000 7f5180c14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:29:53.762 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T05:29:53.763 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:29:53.763 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T05:29:53.763 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:29:53.763 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:29:53.763 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:29:53.763 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:29:53.763 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:29:53.764 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:29:53.764 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:29:54.154 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:29:54.757 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:29:54.756+0000 7f5180c14640 -1 Falling back to public interface 2024-09-16T05:29:55.156 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:29:55.156 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:29:55.156 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:29:55.157 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:29:55.157 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:29:55.157 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:29:55.572 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:29:56.577 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:29:56.577 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:29:56.578 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:29:56.578 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:29:56.578 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:29:56.578 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:29:56.956 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:29:57.327 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:29:57.326+0000 7f5180c14640 -1 osd.1 0 log_to_monitors true 2024-09-16T05:29:57.957 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:29:57.958 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:29:57.958 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:29:57.958 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:29:57.959 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:29:57.959 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:29:58.409 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:29:59.410 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:29:59.410 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:29:59.411 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:29:59.411 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:29:59.411 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:29:59.411 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:29:59.805 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:30:00.806 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:30:00.806 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:30:00.806 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:30:00.807 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:30:00.807 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:30:00.807 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:30:01.291 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:30:02.293 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:30:02.293 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:30:02.293 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:30:02.294 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:30:02.294 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:30:02.294 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:30:02.690 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:30:03.692 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:30:03.692 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:30:03.692 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:30:03.693 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:30:03.693 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:30:03.693 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:30:04.172 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:30:05.170 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:30:05.170 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:30:05.171 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:30:05.171 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:30:05.171 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:30:05.171 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:30:05.554 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:30:06.556 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:30:06.556 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:30:06.556 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:30:06.556 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:30:06.556 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:30:06.557 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:30:06.947 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:30:07.949 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:30:07.949 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:30:07.949 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:30:07.950 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:30:07.950 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:30:07.950 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:30:08.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:30:09.444 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:30:09.445 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:30:09.445 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:30:09.445 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:30:09.446 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:30:09.446 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:30:10.025 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:30:11.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:30:11.027 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:30:11.027 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:30:11.027 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:30:11.027 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:30:11.028 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:30:11.898 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:30:12.899 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:30:12.900 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:30:12.900 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:30:12.900 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:30:12.900 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:30:12.901 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:30:13.626 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:30:14.627 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:30:14.628 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:30:14.628 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:30:14.628 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:30:14.629 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:30:14.629 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:30:15.249 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:30:16.250 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:30:16.251 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:30:16.251 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:30:16.251 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:30:16.252 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:30:16.252 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:30:16.851 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:30:17.853 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:30:17.853 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:30:17.853 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:30:17.854 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:30:17.854 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:30:17.854 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:30:18.470 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:30:19.471 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:30:19.472 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:30:19.472 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:30:19.472 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:30:19.473 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:30:19.473 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:30:19.897 INFO:tasks.workunit.client.0.smithi178.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/402943616,v1:127.0.0.1:6811/402943616] [v2:127.0.0.1:6812/402943616,v1:127.0.0.1:6813/402943616] exists,up 4a80cc51-7246-4835-ba5c-16cf5d0ecad7 2024-09-16T05:30:19.901 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:30:19.902 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:30:19.902 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:30:19.902 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:119: TEST_network_ping_test2: run_osd td/network-ping 2 2024-09-16T05:30:19.902 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-09-16T05:30:19.902 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:30:19.902 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T05:30:19.902 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:30:19.902 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/2 2024-09-16T05:30:19.902 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ad08e9ec-e980-42b8-a051-49ca1d64766e --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-09-16T05:30:19.902 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:30:19.902 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:30:19.902 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:30:19.903 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/2' 2024-09-16T05:30:19.903 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/2/journal' 2024-09-16T05:30:19.903 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:30:19.903 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2024-09-16T05:30:19.903 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-09-16T05:30:19.903 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:30:19.903 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:30:19.903 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:30:19.903 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:30:19.903 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:30:19.903 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41120 2024-09-16T05:30:19.903 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.41120/$cluster-$name.asok' 2024-09-16T05:30:19.903 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.41120/$cluster-$name.asok' 2024-09-16T05:30:19.903 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:30:19.904 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:30:19.904 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:30:19.904 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-09-16T05:30:19.904 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-09-16T05:30:19.904 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:30:19.904 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:30:19.904 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:30:19.904 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:30:19.904 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:30:19.904 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:30:19.904 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/2 2024-09-16T05:30:19.904 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:30:19.905 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=42d28e9c-e9f9-4cbe-b332-aea708ba81f4 2024-09-16T05:30:19.905 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 42d28e9c-e9f9-4cbe-b332-aea708ba81f4' 2024-09-16T05:30:19.905 INFO:tasks.workunit.client.0.smithi178.stdout:add osd2 42d28e9c-e9f9-4cbe-b332-aea708ba81f4 2024-09-16T05:30:19.905 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:30:19.920 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBrwudmMpbRNhAAbxIEpD1TO4hzc927mCZk8g== 2024-09-16T05:30:19.921 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBrwudmMpbRNhAAbxIEpD1TO4hzc927mCZk8g=="}' 2024-09-16T05:30:19.921 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 42d28e9c-e9f9-4cbe-b332-aea708ba81f4 -i td/network-ping/2/new.json 2024-09-16T05:30:20.507 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:30:20.518 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/2/new.json 2024-09-16T05:30:20.519 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=ad08e9ec-e980-42b8-a051-49ca1d64766e --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41120/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBrwudmMpbRNhAAbxIEpD1TO4hzc927mCZk8g== --osd-uuid 42d28e9c-e9f9-4cbe-b332-aea708ba81f4 2024-09-16T05:30:20.549 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:30:20.547+0000 7fdb34687640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:30:20.570 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:30:20.568+0000 7fdb34687640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:30:20.571 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:30:20.569+0000 7fdb34687640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:30:20.571 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:30:20.570+0000 7fdb34687640 -1 bdev(0x55f25942dc00 td/network-ping/2/block) open stat got: (1) Operation not permitted 2024-09-16T05:30:20.572 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:30:20.570+0000 7fdb34687640 -1 bluestore(td/network-ping/2) _read_fsid unparsable uuid 2024-09-16T05:30:26.273 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/2/keyring 2024-09-16T05:30:26.274 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:30:26.274 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd2 key to auth repository 2024-09-16T05:30:26.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T05:30:26.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:30:26.797 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.2 2024-09-16T05:30:26.798 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T05:30:26.799 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=ad08e9ec-e980-42b8-a051-49ca1d64766e --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.41120/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:30:26.799 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:30:26.799 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:30:26.799 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:30:26.825 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:30:26.825+0000 7f00b889c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:30:26.827 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:30:26.827+0000 7f00b889c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:30:26.828 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:30:26.828+0000 7f00b889c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:30:27.170 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T05:30:27.170 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:30:27.170 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T05:30:27.171 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:30:27.171 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:30:27.171 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:30:27.171 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:30:27.172 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:30:27.172 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:30:27.172 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:30:27.541 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:30:28.411 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:30:28.410+0000 7f00b889c640 -1 Falling back to public interface 2024-09-16T05:30:28.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:30:28.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:30:28.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:30:28.543 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:30:28.544 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:30:28.544 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:30:28.961 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:30:29.963 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:30:29.963 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:30:29.963 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:30:29.964 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:30:29.964 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:30:29.964 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:30:30.381 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:30:31.294 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:30:31.293+0000 7f00b889c640 -1 osd.2 0 log_to_monitors true 2024-09-16T05:30:31.382 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:30:31.382 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:30:31.382 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:30:31.383 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:30:31.383 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:30:31.383 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:30:31.949 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:30:32.949 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:30:32.949 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:30:32.950 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:30:32.950 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:30:32.950 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:30:32.950 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:30:33.338 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:30:34.340 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:30:34.340 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:30:34.340 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:30:34.341 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:30:34.341 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:30:34.341 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:30:34.813 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:30:35.815 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:30:35.815 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:30:35.815 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:30:35.816 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:30:35.816 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:30:35.816 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:30:36.267 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:30:37.269 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:30:37.269 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:30:37.269 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:30:37.269 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:30:37.270 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:30:37.270 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:30:37.697 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:30:38.699 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:30:38.699 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:30:38.699 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:30:38.700 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:30:38.700 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:30:38.700 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:30:39.205 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:30:40.206 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:30:40.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:30:40.207 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:30:40.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:30:40.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:30:40.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:30:40.629 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:30:41.630 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:30:41.631 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:30:41.631 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:30:41.631 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:30:41.631 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:30:41.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:30:42.018 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:30:43.020 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:30:43.020 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:30:43.020 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:30:43.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:30:43.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:30:43.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:30:43.582 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:30:44.584 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:30:44.584 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:30:44.584 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:30:44.585 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:30:44.585 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:30:44.585 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:30:45.360 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:30:46.362 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:30:46.362 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:30:46.362 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:30:46.363 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:30:46.363 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:30:46.363 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:30:47.250 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:30:48.251 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:30:48.251 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:30:48.252 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:30:48.252 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:30:48.252 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:30:48.252 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:30:49.286 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:30:50.287 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:30:50.287 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:30:50.288 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:30:50.288 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:30:50.288 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:30:50.288 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:30:51.098 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:30:52.099 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:30:52.100 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:30:52.100 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:30:52.101 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:30:52.101 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:30:52.101 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:30:52.884 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:30:53.885 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:30:53.886 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:30:53.886 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:30:53.886 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:30:53.887 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:30:53.887 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:30:54.273 INFO:tasks.workunit.client.0.smithi178.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2710211181,v1:127.0.0.1:6819/2710211181] [v2:127.0.0.1:6820/2710211181,v1:127.0.0.1:6821/2710211181] exists,up 42d28e9c-e9f9-4cbe-b332-aea708ba81f4 2024-09-16T05:30:54.274 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:30:54.274 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:30:54.274 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:30:54.274 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:121: TEST_network_ping_test2: sleep 5 2024-09-16T05:30:59.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:122: TEST_network_ping_test2: ceph osd crush add-bucket dc1 datacenter 2024-09-16T05:30:59.743 INFO:tasks.workunit.client.0.smithi178.stderr:bucket 'dc1' already exists 2024-09-16T05:30:59.754 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:123: TEST_network_ping_test2: ceph osd crush add-bucket dc2 datacenter 2024-09-16T05:31:00.335 INFO:tasks.workunit.client.0.smithi178.stderr:bucket 'dc2' already exists 2024-09-16T05:31:00.346 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:124: TEST_network_ping_test2: ceph osd crush add-bucket dc3 datacenter 2024-09-16T05:31:00.945 INFO:tasks.workunit.client.0.smithi178.stderr:bucket 'dc3' already exists 2024-09-16T05:31:00.955 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:125: TEST_network_ping_test2: ceph osd crush add-bucket rack1 rack 2024-09-16T05:31:01.539 INFO:tasks.workunit.client.0.smithi178.stderr:bucket 'rack1' already exists 2024-09-16T05:31:01.550 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:126: TEST_network_ping_test2: ceph osd crush add-bucket rack2 rack 2024-09-16T05:31:02.064 INFO:tasks.workunit.client.0.smithi178.stderr:bucket 'rack2' already exists 2024-09-16T05:31:02.075 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:127: TEST_network_ping_test2: ceph osd crush add-bucket rack3 rack 2024-09-16T05:31:02.695 INFO:tasks.workunit.client.0.smithi178.stderr:bucket 'rack3' already exists 2024-09-16T05:31:02.706 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:128: TEST_network_ping_test2: ceph osd crush add-bucket host1 host 2024-09-16T05:31:03.239 INFO:tasks.workunit.client.0.smithi178.stderr:bucket 'host1' already exists 2024-09-16T05:31:03.250 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:129: TEST_network_ping_test2: ceph osd crush add-bucket host2 host 2024-09-16T05:31:03.817 INFO:tasks.workunit.client.0.smithi178.stderr:bucket 'host2' already exists 2024-09-16T05:31:03.828 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:130: TEST_network_ping_test2: ceph osd crush add-bucket host3 host 2024-09-16T05:31:04.299 INFO:tasks.workunit.client.0.smithi178.stderr:bucket 'host3' already exists 2024-09-16T05:31:04.312 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:131: TEST_network_ping_test2: ceph osd crush move dc1 root=default 2024-09-16T05:31:04.947 INFO:tasks.workunit.client.0.smithi178.stderr:no need to move item id -5 name 'dc1' to location {root=default} in crush map 2024-09-16T05:31:04.958 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:132: TEST_network_ping_test2: ceph osd crush move dc2 root=default 2024-09-16T05:31:05.567 INFO:tasks.workunit.client.0.smithi178.stderr:no need to move item id -6 name 'dc2' to location {root=default} in crush map 2024-09-16T05:31:05.578 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:133: TEST_network_ping_test2: ceph osd crush move dc3 root=default 2024-09-16T05:31:06.084 INFO:tasks.workunit.client.0.smithi178.stderr:no need to move item id -7 name 'dc3' to location {root=default} in crush map 2024-09-16T05:31:06.095 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:134: TEST_network_ping_test2: ceph osd crush move rack1 datacenter=dc1 2024-09-16T05:31:06.681 INFO:tasks.workunit.client.0.smithi178.stderr:no need to move item id -8 name 'rack1' to location {datacenter=dc1} in crush map 2024-09-16T05:31:06.691 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:135: TEST_network_ping_test2: ceph osd crush move rack2 datacenter=dc2 2024-09-16T05:31:07.114 INFO:tasks.workunit.client.0.smithi178.stderr:no need to move item id -9 name 'rack2' to location {datacenter=dc2} in crush map 2024-09-16T05:31:07.125 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:136: TEST_network_ping_test2: ceph osd crush move rack3 datacenter=dc3 2024-09-16T05:31:07.612 INFO:tasks.workunit.client.0.smithi178.stderr:no need to move item id -10 name 'rack3' to location {datacenter=dc3} in crush map 2024-09-16T05:31:07.623 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:137: TEST_network_ping_test2: ceph osd crush move host1 rack=rack1 2024-09-16T05:31:08.111 INFO:tasks.workunit.client.0.smithi178.stderr:no need to move item id -11 name 'host1' to location {rack=rack1} in crush map 2024-09-16T05:31:08.121 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:138: TEST_network_ping_test2: ceph osd crush move host2 rack=rack2 2024-09-16T05:31:08.637 INFO:tasks.workunit.client.0.smithi178.stderr:no need to move item id -12 name 'host2' to location {rack=rack2} in crush map 2024-09-16T05:31:08.648 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:139: TEST_network_ping_test2: ceph osd crush move host3 rack=rack3 2024-09-16T05:31:09.140 INFO:tasks.workunit.client.0.smithi178.stderr:no need to move item id -13 name 'host3' to location {rack=rack3} in crush map 2024-09-16T05:31:09.151 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:140: TEST_network_ping_test2: ceph osd crush set osd.0 1.0 host=host1 2024-09-16T05:31:09.633 INFO:tasks.workunit.client.0.smithi178.stderr:set item id 0 name 'osd.0' weight 1 at location {host=host1}: no change 2024-09-16T05:31:09.644 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:141: TEST_network_ping_test2: ceph osd crush set osd.1 1.0 host=host2 2024-09-16T05:31:10.177 INFO:tasks.workunit.client.0.smithi178.stderr:set item id 1 name 'osd.1' weight 1 at location {host=host2}: no change 2024-09-16T05:31:10.188 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:142: TEST_network_ping_test2: ceph osd crush set osd.2 1.0 host=host3 2024-09-16T05:31:10.758 INFO:tasks.workunit.client.0.smithi178.stderr:set item id 2 name 'osd.2' weight 1 at location {host=host3}: no change 2024-09-16T05:31:10.769 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:143: TEST_network_ping_test2: ceph osd crush rule create-simple myrule default host firstn 2024-09-16T05:31:11.302 INFO:tasks.workunit.client.0.smithi178.stderr:rule myrule already exists 2024-09-16T05:31:11.313 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:145: TEST_network_ping_test2: create_pool foo 16 16 replicated myrule 2024-09-16T05:31:11.313 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 16 replicated myrule 2024-09-16T05:31:12.318 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'foo' already exists 2024-09-16T05:31:12.331 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T05:31:13.331 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:148: TEST_network_ping_test2: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2024-09-16T05:31:13.397 INFO:tasks.workunit.client.0.smithi178.stdout:hints = 1 2024-09-16T05:31:13.397 INFO:tasks.workunit.client.0.smithi178.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2024-09-16T05:31:13.397 INFO:tasks.workunit.client.0.smithi178.stdout:Object prefix: benchmark_data_smithi178_53517 2024-09-16T05:31:13.397 INFO:tasks.workunit.client.0.smithi178.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T05:31:13.397 INFO:tasks.workunit.client.0.smithi178.stdout: 0 0 0 0 0 0 - 0 2024-09-16T05:31:14.395 INFO:tasks.workunit.client.0.smithi178.stdout: 1 16 58 42 0.164043 0.164062 0.340565 0.288047 2024-09-16T05:31:15.395 INFO:tasks.workunit.client.0.smithi178.stdout: 2 16 136 120 0.234349 0.304688 0.227924 0.259247 2024-09-16T05:31:16.396 INFO:tasks.workunit.client.0.smithi178.stdout: 3 16 200 184 0.239554 0.25 0.248339 0.253257 2024-09-16T05:31:17.396 INFO:tasks.workunit.client.0.smithi178.stdout: 4 16 265 249 0.243133 0.253906 0.243188 0.247081 2024-09-16T05:31:18.397 INFO:tasks.workunit.client.0.smithi178.stdout: 5 16 344 328 0.25618 0.308594 0.233488 0.24348 2024-09-16T05:31:19.397 INFO:tasks.workunit.client.0.smithi178.stdout: 6 16 408 392 0.255145 0.25 0.206935 0.241351 2024-09-16T05:31:20.397 INFO:tasks.workunit.client.0.smithi178.stdout: 7 16 472 456 0.254406 0.25 0.24148 0.242084 2024-09-16T05:31:21.397 INFO:tasks.workunit.client.0.smithi178.stdout: 8 16 537 521 0.254339 0.253906 0.200282 0.240746 2024-09-16T05:31:22.397 INFO:tasks.workunit.client.0.smithi178.stdout: 9 16 618 602 0.261229 0.316406 0.190435 0.235351 2024-09-16T05:31:23.397 INFO:tasks.workunit.client.0.smithi178.stdout: 10 16 681 665 0.259712 0.246094 0.184308 0.237216 2024-09-16T05:31:24.398 INFO:tasks.workunit.client.0.smithi178.stdout:Total time run: 10.1061 2024-09-16T05:31:24.398 INFO:tasks.workunit.client.0.smithi178.stdout:Total writes made: 681 2024-09-16T05:31:24.398 INFO:tasks.workunit.client.0.smithi178.stdout:Write size: 4096 2024-09-16T05:31:24.398 INFO:tasks.workunit.client.0.smithi178.stdout:Object size: 4096 2024-09-16T05:31:24.398 INFO:tasks.workunit.client.0.smithi178.stdout:Bandwidth (MB/sec): 0.263222 2024-09-16T05:31:24.398 INFO:tasks.workunit.client.0.smithi178.stdout:Stddev Bandwidth: 0.0438572 2024-09-16T05:31:24.398 INFO:tasks.workunit.client.0.smithi178.stdout:Max bandwidth (MB/sec): 0.316406 2024-09-16T05:31:24.398 INFO:tasks.workunit.client.0.smithi178.stdout:Min bandwidth (MB/sec): 0.164062 2024-09-16T05:31:24.398 INFO:tasks.workunit.client.0.smithi178.stdout:Average IOPS: 67 2024-09-16T05:31:24.399 INFO:tasks.workunit.client.0.smithi178.stdout:Stddev IOPS: 11.2274 2024-09-16T05:31:24.399 INFO:tasks.workunit.client.0.smithi178.stdout:Max IOPS: 81 2024-09-16T05:31:24.399 INFO:tasks.workunit.client.0.smithi178.stdout:Min IOPS: 42 2024-09-16T05:31:24.399 INFO:tasks.workunit.client.0.smithi178.stdout:Average Latency(s): 0.23575 2024-09-16T05:31:24.399 INFO:tasks.workunit.client.0.smithi178.stdout:Stddev Latency(s): 0.0513353 2024-09-16T05:31:24.399 INFO:tasks.workunit.client.0.smithi178.stdout:Max latency(s): 0.470221 2024-09-16T05:31:24.399 INFO:tasks.workunit.client.0.smithi178.stdout:Min latency(s): 0.156694 2024-09-16T05:31:24.472 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:151: TEST_network_ping_test2: sleep 10 2024-09-16T05:31:34.474 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:152: TEST_network_ping_test2: flush_pg_stats 2024-09-16T05:31:34.474 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-16T05:31:34.475 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-16T05:31:34.855 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-16T05:31:34.855 INFO:tasks.workunit.client.0.smithi178.stderr:1 2024-09-16T05:31:34.855 INFO:tasks.workunit.client.0.smithi178.stderr:2' 2024-09-16T05:31:34.855 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-16T05:31:34.855 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:31:34.855 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T05:31:34.968 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836508 2024-09-16T05:31:34.968 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836508 2024-09-16T05:31:34.968 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508' 2024-09-16T05:31:34.969 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:31:34.969 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T05:31:35.088 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672983 2024-09-16T05:31:35.088 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672983 2024-09-16T05:31:35.089 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672983' 2024-09-16T05:31:35.089 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:31:35.089 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T05:31:35.201 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509461 2024-09-16T05:31:35.201 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509461 2024-09-16T05:31:35.201 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672983 2-64424509461' 2024-09-16T05:31:35.201 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:31:35.201 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836508 2024-09-16T05:31:35.201 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:31:35.202 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-16T05:31:35.202 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836508 2024-09-16T05:31:35.203 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:31:35.203 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836508 2024-09-16T05:31:35.204 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.0 seq 21474836508 2024-09-16T05:31:35.204 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-09-16T05:31:35.204 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:31:35.614 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836508 2024-09-16T05:31:35.615 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-16T05:31:36.615 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T05:31:36.616 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:31:37.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836509 -lt 21474836508 2024-09-16T05:31:37.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:31:37.042 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672983 2024-09-16T05:31:37.042 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:31:37.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-09-16T05:31:37.043 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672983 2024-09-16T05:31:37.043 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:31:37.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672983 2024-09-16T05:31:37.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-09-16T05:31:37.045 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.1 seq 42949672983 2024-09-16T05:31:37.045 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T05:31:37.414 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672983 -lt 42949672983 2024-09-16T05:31:37.414 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:31:37.415 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509461 2024-09-16T05:31:37.415 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:31:37.415 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-09-16T05:31:37.416 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509461 2024-09-16T05:31:37.416 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:31:37.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509461 2024-09-16T05:31:37.417 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.2 seq 64424509461 2024-09-16T05:31:37.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-09-16T05:31:37.417 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T05:31:37.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509461 -lt 64424509461 2024-09-16T05:31:37.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:154: TEST_network_ping_test2: ceph health 2024-09-16T05:31:37.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:154: TEST_network_ping_test2: tee td/network-ping/health 2024-09-16T05:31:38.268 INFO:tasks.workunit.client.0.smithi178.stdout:HEALTH_WARN Slow OSD heartbeats on back (longest 0.362ms); Slow OSD heartbeats on front (longest 0.373ms); 1 pool(s) do not have an application enabled 2024-09-16T05:31:38.278 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:155: TEST_network_ping_test2: grep -q 'Slow OSD heartbeats' td/network-ping/health 2024-09-16T05:31:38.280 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:157: TEST_network_ping_test2: ceph health detail 2024-09-16T05:31:38.280 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:157: TEST_network_ping_test2: tee td/network-ping/health 2024-09-16T05:31:38.773 INFO:tasks.workunit.client.0.smithi178.stdout:HEALTH_WARN Slow OSD heartbeats on back (longest 0.362ms); Slow OSD heartbeats on front (longest 0.373ms); 1 pool(s) do not have an application enabled 2024-09-16T05:31:38.773 INFO:tasks.workunit.client.0.smithi178.stdout:[WRN] OSD_SLOW_PING_TIME_BACK: Slow OSD heartbeats on back (longest 0.362ms) 2024-09-16T05:31:38.773 INFO:tasks.workunit.client.0.smithi178.stdout: Slow OSD heartbeats on back from osd.0 [dc1,rack1] to osd.2 [dc3,rack3] 0.362 msec 2024-09-16T05:31:38.773 INFO:tasks.workunit.client.0.smithi178.stdout: Slow OSD heartbeats on back from osd.2 [dc3,rack3] to osd.1 [dc2,rack2] 0.332 msec 2024-09-16T05:31:38.773 INFO:tasks.workunit.client.0.smithi178.stdout: Slow OSD heartbeats on back from osd.1 [dc2,rack2] to osd.2 [dc3,rack3] 0.327 msec 2024-09-16T05:31:38.774 INFO:tasks.workunit.client.0.smithi178.stdout: Slow OSD heartbeats on back from osd.2 [dc3,rack3] to osd.0 [dc1,rack1] 0.312 msec 2024-09-16T05:31:38.774 INFO:tasks.workunit.client.0.smithi178.stdout: Slow OSD heartbeats on back from osd.0 [dc1,rack1] to osd.1 [dc2,rack2] 0.284 msec 2024-09-16T05:31:38.774 INFO:tasks.workunit.client.0.smithi178.stdout: Slow OSD heartbeats on back from osd.1 [dc2,rack2] to osd.0 [dc1,rack1] 0.278 msec 2024-09-16T05:31:38.774 INFO:tasks.workunit.client.0.smithi178.stdout:[WRN] OSD_SLOW_PING_TIME_FRONT: Slow OSD heartbeats on front (longest 0.373ms) 2024-09-16T05:31:38.774 INFO:tasks.workunit.client.0.smithi178.stdout: Slow OSD heartbeats on front from osd.0 [dc1,rack1] to osd.2 [dc3,rack3] 0.373 msec 2024-09-16T05:31:38.774 INFO:tasks.workunit.client.0.smithi178.stdout: Slow OSD heartbeats on front from osd.2 [dc3,rack3] to osd.1 [dc2,rack2] 0.355 msec 2024-09-16T05:31:38.774 INFO:tasks.workunit.client.0.smithi178.stdout: Slow OSD heartbeats on front from osd.2 [dc3,rack3] to osd.0 [dc1,rack1] 0.342 msec 2024-09-16T05:31:38.774 INFO:tasks.workunit.client.0.smithi178.stdout: Slow OSD heartbeats on front from osd.1 [dc2,rack2] to osd.0 [dc1,rack1] 0.335 msec 2024-09-16T05:31:38.774 INFO:tasks.workunit.client.0.smithi178.stdout: Slow OSD heartbeats on front from osd.1 [dc2,rack2] to osd.2 [dc3,rack3] 0.324 msec 2024-09-16T05:31:38.774 INFO:tasks.workunit.client.0.smithi178.stdout: Slow OSD heartbeats on front from osd.0 [dc1,rack1] to osd.1 [dc2,rack2] 0.296 msec 2024-09-16T05:31:38.774 INFO:tasks.workunit.client.0.smithi178.stdout:[WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-16T05:31:38.774 INFO:tasks.workunit.client.0.smithi178.stdout: application not enabled on pool 'foo' 2024-09-16T05:31:38.774 INFO:tasks.workunit.client.0.smithi178.stdout: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-16T05:31:38.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:158: TEST_network_ping_test2: grep -q OSD_SLOW_PING_TIME_BACK td/network-ping/health 2024-09-16T05:31:38.785 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:159: TEST_network_ping_test2: grep -q OSD_SLOW_PING_TIME_FRONT td/network-ping/health 2024-09-16T05:31:38.786 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:160: TEST_network_ping_test2: grep -q 'Slow OSD heartbeats on front from osd[.][0-2] [[]dc[1-3],rack[1-3][]] to osd[.][0-2] [[]dc[1-3],rack[1-3][]]' td/network-ping/health 2024-09-16T05:31:38.787 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:162: TEST_network_ping_test2: rm -f td/network-ping/health 2024-09-16T05:31:38.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:20: run: teardown td/network-ping 2024-09-16T05:31:38.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-09-16T05:31:38.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:31:38.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-09-16T05:31:38.789 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:31:38.789 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:31:38.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:31:38.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:31:38.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:31:38.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:31:38.896 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:31:38.897 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:31:38.897 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:31:38.898 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:31:38.898 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:31:38.898 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:31:38.899 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:31:38.899 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:31:38.900 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:31:38.900 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:31:38.900 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:31:38.901 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:31:38.901 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-09-16T05:31:38.922 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:31:38.922 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:31:38.922 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41120 2024-09-16T05:31:38.922 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.41120 2024-09-16T05:31:38.923 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:31:38.923 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:31:38.923 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-09-16T05:31:38.924 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/network-ping 0 2024-09-16T05:31:38.924 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-09-16T05:31:38.924 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-16T05:31:38.924 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-09-16T05:31:38.924 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:31:38.924 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:31:38.924 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:31:38.924 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:31:38.924 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:31:38.925 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:31:38.925 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:31:38.925 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:31:38.926 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:31:38.927 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:31:38.927 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:31:38.927 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:31:38.928 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:31:38.928 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:31:38.928 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:31:38.929 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:31:38.929 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:31:38.930 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-16T05:31:38.930 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-09-16T05:31:38.931 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:31:38.931 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:31:38.931 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.41120 2024-09-16T05:31:38.931 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.41120 2024-09-16T05:31:38.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:31:38.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:31:38.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-09-16T05:31:38.933 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T05:31:38.934 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T05:31:38.960 INFO:tasks.workunit:Running workunit misc/ok-to-stop.sh... 2024-09-16T05:31:38.960 DEBUG:teuthology.orchestra.run.smithi178:workunit test misc/ok-to-stop.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ad35db06e8924912f01e32b35576297c0f1352b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh 2024-09-16T05:31:39.022 INFO:tasks.workunit.client.0.smithi178.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-16T05:31:39.029 INFO:tasks.workunit.client.0.smithi178.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-16T05:31:39.029 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-16T05:31:39.029 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-16T05:31:39.029 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-09-16T05:31:39.029 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-09-16T05:31:39.029 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-09-16T05:31:39.029 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-09-16T05:31:39.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-09-16T05:31:39.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-09-16T05:31:39.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/ok-to-stop 2024-09-16T05:31:39.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:6: run: local dir=td/ok-to-stop 2024-09-16T05:31:39.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:7: run: shift 2024-09-16T05:31:39.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:9: run: export CEPH_MON_A=127.0.0.1:7150 2024-09-16T05:31:39.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:9: run: CEPH_MON_A=127.0.0.1:7150 2024-09-16T05:31:39.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:10: run: export CEPH_MON_B=127.0.0.1:7151 2024-09-16T05:31:39.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:10: run: CEPH_MON_B=127.0.0.1:7151 2024-09-16T05:31:39.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:11: run: export CEPH_MON_C=127.0.0.1:7152 2024-09-16T05:31:39.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:11: run: CEPH_MON_C=127.0.0.1:7152 2024-09-16T05:31:39.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:12: run: export CEPH_MON_D=127.0.0.1:7153 2024-09-16T05:31:39.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:12: run: CEPH_MON_D=127.0.0.1:7153 2024-09-16T05:31:39.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:13: run: export CEPH_MON_E=127.0.0.1:7154 2024-09-16T05:31:39.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:13: run: CEPH_MON_E=127.0.0.1:7154 2024-09-16T05:31:39.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:14: run: export CEPH_ARGS 2024-09-16T05:31:39.031 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:15: run: uuidgen 2024-09-16T05:31:39.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:15: run: CEPH_ARGS+='--fsid=018ac118-e789-4c01-be89-78aa14fbe62f --auth-supported=none ' 2024-09-16T05:31:39.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:16: run: export 'ORIG_CEPH_ARGS=--fsid=018ac118-e789-4c01-be89-78aa14fbe62f --auth-supported=none ' 2024-09-16T05:31:39.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:16: run: ORIG_CEPH_ARGS='--fsid=018ac118-e789-4c01-be89-78aa14fbe62f --auth-supported=none ' 2024-09-16T05:31:39.031 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: set 2024-09-16T05:31:39.031 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-16T05:31:39.033 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: local 'funcs=TEST_0_mds 2024-09-16T05:31:39.033 INFO:tasks.workunit.client.0.smithi178.stderr:TEST_0_osd 2024-09-16T05:31:39.033 INFO:tasks.workunit.client.0.smithi178.stderr:TEST_1_mon_checks 2024-09-16T05:31:39.033 INFO:tasks.workunit.client.0.smithi178.stderr:TEST_2_mons_checks 2024-09-16T05:31:39.033 INFO:tasks.workunit.client.0.smithi178.stderr:TEST_3_mons_checks 2024-09-16T05:31:39.033 INFO:tasks.workunit.client.0.smithi178.stderr:TEST_4_mons_checks 2024-09-16T05:31:39.033 INFO:tasks.workunit.client.0.smithi178.stderr:TEST_5_mons_checks' 2024-09-16T05:31:39.033 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-16T05:31:39.033 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-16T05:31:39.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-09-16T05:31:39.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-09-16T05:31:39.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-16T05:31:39.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:31:39.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-16T05:31:39.034 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:31:39.034 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:31:39.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:31:39.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:31:39.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:31:39.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:31:39.034 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:31:39.035 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:31:39.036 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:31:39.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:31:39.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:31:39.036 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:31:39.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:31:39.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:31:39.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:31:39.036 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:31:39.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:31:39.037 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:31:39.037 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-16T05:31:39.039 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:31:39.039 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:31:39.039 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:31:39.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53882 2024-09-16T05:31:39.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:31:39.040 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:31:39.040 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-09-16T05:31:39.040 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:31:39.040 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:31:39.041 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:31:39.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.53882 2024-09-16T05:31:39.042 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:31:39.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-16T05:31:39.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-16T05:31:39.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:31:39.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-16T05:31:39.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_0_mds td/ok-to-stop 2024-09-16T05:31:39.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:208: TEST_0_mds: local dir=td/ok-to-stop 2024-09-16T05:31:39.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:210: TEST_0_mds: CEPH_ARGS='--fsid=018ac118-e789-4c01-be89-78aa14fbe62f --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-16T05:31:39.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:212: TEST_0_mds: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-16T05:31:39.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-16T05:31:39.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:31:39.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:31:39.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:31:39.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-09-16T05:31:39.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-16T05:31:39.402 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:31:39.402 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:31:39.403 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:31:39.403 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:31:39.403 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:31:39.403 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:31:39.403 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53882/$cluster-$name.asok' 2024-09-16T05:31:39.403 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53882/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-16T05:31:39.586 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:31:39.588 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:31:39.588 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:31:39.588 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:31:39.588 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:31:39.588 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:31:39.588 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:31:39.588 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:31:39.589 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:31:39.589 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:31:39.589 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:31:39.589 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:31:39.589 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53882/ceph-mon.a.asok 2024-09-16T05:31:39.589 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:31:39.589 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53882/ceph-mon.a.asok config get fsid 2024-09-16T05:31:39.668 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:31:39.668 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:31:39.668 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:31:39.668 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:31:39.668 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:31:39.668 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:31:39.668 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:31:39.668 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:31:39.668 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:31:39.669 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:31:39.669 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:31:39.669 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53882/ceph-mon.a.asok 2024-09-16T05:31:39.669 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:31:39.669 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53882/ceph-mon.a.asok config get mon_host 2024-09-16T05:31:39.745 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:213: TEST_0_mds: run_mgr td/ok-to-stop x 2024-09-16T05:31:39.746 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ok-to-stop 2024-09-16T05:31:39.746 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T05:31:39.746 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T05:31:39.746 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T05:31:39.746 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ok-to-stop/x 2024-09-16T05:31:39.746 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T05:31:40.587 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T05:31:40.587 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:31:40.588 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:31:40.588 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:31:40.588 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:31:40.588 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:31:40.588 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53882/$cluster-$name.asok' 2024-09-16T05:31:40.588 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:31:40.588 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-stop/x '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53882/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:31:40.616 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:214: TEST_0_mds: run_osd td/ok-to-stop 0 2024-09-16T05:31:40.616 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-09-16T05:31:40.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:31:40.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T05:31:40.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:31:40.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/0 2024-09-16T05:31:40.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=018ac118-e789-4c01-be89-78aa14fbe62f --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-16T05:31:40.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:31:40.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:31:40.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:31:40.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/0' 2024-09-16T05:31:40.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/0/journal' 2024-09-16T05:31:40.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:31:40.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:31:40.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-09-16T05:31:40.618 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:31:40.618 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:31:40.618 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:31:40.618 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:31:40.618 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:31:40.618 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:31:40.618 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53882/$cluster-$name.asok' 2024-09-16T05:31:40.618 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53882/$cluster-$name.asok' 2024-09-16T05:31:40.618 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:31:40.618 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:31:40.618 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:31:40.618 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-09-16T05:31:40.618 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-09-16T05:31:40.618 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:31:40.619 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:31:40.619 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:31:40.619 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:31:40.619 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:31:40.619 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:31:40.619 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/0 2024-09-16T05:31:40.619 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:31:40.619 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f5df3c7a-7db8-4356-bb3a-42df153fb54f 2024-09-16T05:31:40.620 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f5df3c7a-7db8-4356-bb3a-42df153fb54f' 2024-09-16T05:31:40.620 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 f5df3c7a-7db8-4356-bb3a-42df153fb54f 2024-09-16T05:31:40.620 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:31:40.634 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC8wudmLMy+JRAA/nh+IIVUL3eXvFEAku3GJg== 2024-09-16T05:31:40.634 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC8wudmLMy+JRAA/nh+IIVUL3eXvFEAku3GJg=="}' 2024-09-16T05:31:40.634 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f5df3c7a-7db8-4356-bb3a-42df153fb54f -i td/ok-to-stop/0/new.json 2024-09-16T05:31:40.908 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:31:40.916 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/0/new.json 2024-09-16T05:31:40.917 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=018ac118-e789-4c01-be89-78aa14fbe62f --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53882/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC8wudmLMy+JRAA/nh+IIVUL3eXvFEAku3GJg== --osd-uuid f5df3c7a-7db8-4356-bb3a-42df153fb54f 2024-09-16T05:31:40.944 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:31:40.943+0000 7f0d86612640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:31:40.976 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:31:40.975+0000 7f0d86612640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:31:40.977 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:31:40.976+0000 7f0d86612640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:31:40.977 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:31:40.977+0000 7f0d86612640 -1 bdev(0x55f2b0ffdc00 td/ok-to-stop/0/block) open stat got: (1) Operation not permitted 2024-09-16T05:31:40.977 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:31:40.977+0000 7f0d86612640 -1 bluestore(td/ok-to-stop/0) _read_fsid unparsable uuid 2024-09-16T05:31:46.712 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/0/keyring 2024-09-16T05:31:46.712 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:31:46.713 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T05:31:46.713 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T05:31:46.713 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:31:47.220 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:31:47.221 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T05:31:47.221 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=018ac118-e789-4c01-be89-78aa14fbe62f --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53882/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:31:47.221 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:31:47.221 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:31:47.222 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:31:47.244 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:31:47.243+0000 7f2965934640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:31:47.271 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:31:47.270+0000 7f2965934640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:31:47.272 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:31:47.272+0000 7f2965934640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:31:47.633 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T05:31:47.633 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:31:47.633 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:31:47.633 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:31:47.633 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:31:47.633 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:31:47.633 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:31:47.634 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:31:47.634 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:31:47.634 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:31:48.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:31:48.609 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:31:48.608+0000 7f2965934640 -1 Falling back to public interface 2024-09-16T05:31:49.045 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:31:49.045 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:31:49.045 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:31:49.046 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:31:49.046 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:31:49.046 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:31:49.461 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:31:50.463 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:31:50.463 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:31:50.463 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:31:50.463 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:31:50.464 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:31:50.464 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:31:50.912 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:31:51.399 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:31:51.398+0000 7f2965934640 -1 osd.0 0 log_to_monitors true 2024-09-16T05:31:51.914 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:31:51.914 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:31:51.914 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:31:51.915 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:31:51.915 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:31:51.915 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:31:52.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:31:53.444 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:31:53.445 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:31:53.445 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:31:53.445 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:31:53.445 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:31:53.445 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:31:53.836 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:31:54.837 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:31:54.838 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:31:54.838 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:31:54.838 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:31:54.839 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:31:54.839 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:31:55.271 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:31:56.272 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:31:56.273 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:31:56.273 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:31:56.273 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:31:56.273 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:31:56.274 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:31:56.678 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:31:57.680 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:31:57.680 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:31:57.680 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:31:57.680 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:31:57.681 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:31:57.681 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:31:58.150 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:31:59.152 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:31:59.152 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:31:59.152 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:31:59.152 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:31:59.153 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:31:59.153 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:31:59.533 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:32:00.535 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:32:00.535 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:32:00.535 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:32:00.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:32:00.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:32:00.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:32:00.934 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:32:01.936 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:32:01.936 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:32:01.936 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:32:01.937 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:32:01.937 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:32:01.937 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:32:02.563 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:32:03.565 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:32:03.565 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:32:03.565 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:32:03.566 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:32:03.566 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:32:03.566 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:32:04.246 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:32:05.247 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:32:05.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:32:05.248 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:32:05.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:32:05.249 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:32:05.249 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:32:06.029 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:32:07.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:32:07.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:32:07.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:32:07.031 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:32:07.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:32:07.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:32:07.859 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:32:08.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:32:08.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:32:08.861 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:32:08.862 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:32:08.862 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:32:08.862 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:32:09.590 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:32:10.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:32:10.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:32:10.593 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:32:10.594 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:32:10.594 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:32:10.594 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:32:11.696 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:32:12.697 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:32:12.697 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:32:12.698 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:32:12.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:32:12.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:32:12.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:32:13.329 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:32:14.331 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:32:14.331 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:32:14.331 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:32:14.332 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:32:14.332 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:32:14.332 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:32:14.750 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/106141269,v1:127.0.0.1:6803/106141269] [v2:127.0.0.1:6804/106141269,v1:127.0.0.1:6805/106141269] exists,up f5df3c7a-7db8-4356-bb3a-42df153fb54f 2024-09-16T05:32:14.752 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:32:14.752 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:32:14.753 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:32:14.753 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:215: TEST_0_mds: run_mds td/ok-to-stop a 2024-09-16T05:32:14.753 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ok-to-stop 2024-09-16T05:32:14.753 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-09-16T05:32:14.753 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=a 2024-09-16T05:32:14.753 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-09-16T05:32:14.753 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ok-to-stop/a 2024-09-16T05:32:14.753 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-09-16T05:32:14.753 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:32:14.753 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:32:14.753 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:32:14.753 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:32:14.753 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:32:14.754 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53882/$cluster-$name.asok' 2024-09-16T05:32:14.754 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id a --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53882/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' 2024-09-16T05:32:14.776 INFO:tasks.workunit.client.0.smithi178.stdout:starting mds.a at 2024-09-16T05:32:14.778 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:217: TEST_0_mds: ceph osd pool create meta 1 2024-09-16T05:32:15.284 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'meta' already exists 2024-09-16T05:32:15.295 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:218: TEST_0_mds: ceph osd pool create data 1 2024-09-16T05:32:15.853 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'data' already exists 2024-09-16T05:32:15.864 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:219: TEST_0_mds: ceph fs new myfs meta data 2024-09-16T05:32:16.797 INFO:tasks.workunit.client.0.smithi178.stderr:filesystem 'myfs' already exists 2024-09-16T05:32:16.808 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:220: TEST_0_mds: sleep 5 2024-09-16T05:32:21.810 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:222: TEST_0_mds: ceph mds ok-to-stop a 2024-09-16T05:32:22.138 INFO:tasks.workunit.client.0.smithi178.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 2024-09-16T05:32:22.142 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:223: TEST_0_mds: ceph mds ok-to-stop a dne 2024-09-16T05:32:22.469 INFO:tasks.workunit.client.0.smithi178.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 2024-09-16T05:32:22.473 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:224: TEST_0_mds: ceph mds ok-to-stop dne 2024-09-16T05:32:22.976 INFO:tasks.workunit.client.0.smithi178.stderr:should be safe to stop [dne] 2024-09-16T05:32:22.988 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:226: TEST_0_mds: run_mds td/ok-to-stop b 2024-09-16T05:32:22.988 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ok-to-stop 2024-09-16T05:32:22.988 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-09-16T05:32:22.988 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=b 2024-09-16T05:32:22.988 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-09-16T05:32:22.988 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ok-to-stop/b 2024-09-16T05:32:22.988 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-09-16T05:32:22.988 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:32:22.988 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:32:22.988 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:32:22.989 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:32:22.989 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:32:22.989 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53882/$cluster-$name.asok' 2024-09-16T05:32:22.989 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id b --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53882/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' 2024-09-16T05:32:23.037 INFO:tasks.workunit.client.0.smithi178.stdout:starting mds.b at 2024-09-16T05:32:23.040 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:227: TEST_0_mds: sleep 5 2024-09-16T05:32:28.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:229: TEST_0_mds: ceph mds ok-to-stop a 2024-09-16T05:32:28.546 INFO:tasks.workunit.client.0.smithi178.stderr:should be safe to stop [a] 2024-09-16T05:32:28.557 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:230: TEST_0_mds: ceph mds ok-to-stop b 2024-09-16T05:32:29.082 INFO:tasks.workunit.client.0.smithi178.stderr:should be safe to stop [b] 2024-09-16T05:32:29.093 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:231: TEST_0_mds: ceph mds ok-to-stop a b 2024-09-16T05:32:29.451 INFO:tasks.workunit.client.0.smithi178.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 4167 2024-09-16T05:32:29.455 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:232: TEST_0_mds: ceph mds ok-to-stop a dne1 dne2 2024-09-16T05:32:29.956 INFO:tasks.workunit.client.0.smithi178.stderr:should be safe to stop [a,dne1,dne2] 2024-09-16T05:32:29.968 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:233: TEST_0_mds: ceph mds ok-to-stop b dne 2024-09-16T05:32:30.499 INFO:tasks.workunit.client.0.smithi178.stderr:should be safe to stop [b,dne] 2024-09-16T05:32:30.510 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:234: TEST_0_mds: ceph mds ok-to-stop a b dne 2024-09-16T05:32:30.868 INFO:tasks.workunit.client.0.smithi178.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 4167 2024-09-16T05:32:30.872 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:235: TEST_0_mds: ceph mds ok-to-stop dne1 dne2 2024-09-16T05:32:31.370 INFO:tasks.workunit.client.0.smithi178.stderr:should be safe to stop [dne1,dne2] 2024-09-16T05:32:31.380 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:237: TEST_0_mds: kill_daemons td/ok-to-stop KILL mds.a 2024-09-16T05:32:31.381 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:32:31.381 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:32:31.381 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:32:31.381 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:32:31.381 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:32:31.488 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:32:31.488 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-16T05:32:31.488 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:32:31.488 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:32:31.488 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:32:31.488 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:32:31.488 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:32:31.594 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:32:31.594 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-16T05:32:31.594 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-16T05:32:31.594 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:32:31.594 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-16T05:32:31.594 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:32:31.594 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:32:31.595 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:32:31.595 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:32:31.595 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:32:31.599 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:32:31.599 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:32:31.600 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:32:31.600 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:32:31.601 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:32:31.601 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:32:31.601 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:32:31.602 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:32:31.602 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:32:31.603 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:32:31.603 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:32:31.603 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:32:31.604 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:32:31.604 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-16T05:32:31.610 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:32:31.611 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:32:31.611 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:32:31.611 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53882 2024-09-16T05:32:31.612 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:32:31.613 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:32:31.613 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-16T05:32:31.613 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-16T05:32:31.613 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-09-16T05:32:31.613 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-09-16T05:32:31.613 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-16T05:32:31.613 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:32:31.613 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-16T05:32:31.614 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:32:31.614 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:32:31.614 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:32:31.614 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:32:31.614 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:32:31.615 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:32:31.615 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:32:31.615 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:32:31.615 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:32:31.616 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:32:31.616 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:32:31.616 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:32:31.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:32:31.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:32:31.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:32:31.617 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:32:31.618 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:32:31.619 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:32:31.619 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-16T05:32:31.620 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:32:31.620 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:32:31.620 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:32:31.620 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53882 2024-09-16T05:32:31.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:32:31.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:32:31.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-09-16T05:32:31.622 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:32:31.622 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:32:31.622 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:32:31.622 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.53882 2024-09-16T05:32:31.624 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:32:31.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:32:31.625 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:32:31.625 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-16T05:32:31.625 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_0_osd td/ok-to-stop 2024-09-16T05:32:31.625 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:241: TEST_0_osd: local dir=td/ok-to-stop 2024-09-16T05:32:31.625 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:243: TEST_0_osd: CEPH_ARGS='--fsid=018ac118-e789-4c01-be89-78aa14fbe62f --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-16T05:32:31.625 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:245: TEST_0_osd: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-16T05:32:31.625 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-16T05:32:31.625 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:32:31.625 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:32:31.625 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:32:31.625 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-09-16T05:32:31.625 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-16T05:32:31.975 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:32:31.976 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:32:31.976 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:32:31.976 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:32:31.976 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:32:31.976 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:32:31.976 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53882/$cluster-$name.asok' 2024-09-16T05:32:31.976 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53882/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-16T05:32:32.168 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:32:32.170 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:32:32.170 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:32:32.170 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:32:32.171 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:32:32.171 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:32:32.171 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:32:32.171 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:32:32.171 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:32:32.171 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:32:32.171 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:32:32.171 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:32:32.171 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53882/ceph-mon.a.asok 2024-09-16T05:32:32.171 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:32:32.171 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53882/ceph-mon.a.asok config get fsid 2024-09-16T05:32:32.250 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:32:32.250 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:32:32.250 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:32:32.250 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:32:32.251 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:32:32.251 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:32:32.251 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:32:32.251 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:32:32.251 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:32:32.251 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:32:32.251 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:32:32.251 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53882/ceph-mon.a.asok 2024-09-16T05:32:32.251 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:32:32.251 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53882/ceph-mon.a.asok config get mon_host 2024-09-16T05:32:32.327 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:246: TEST_0_osd: run_mgr td/ok-to-stop x 2024-09-16T05:32:32.328 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ok-to-stop 2024-09-16T05:32:32.328 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T05:32:32.328 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T05:32:32.328 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T05:32:32.328 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ok-to-stop/x 2024-09-16T05:32:32.328 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T05:32:33.173 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T05:32:33.174 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:32:33.174 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:32:33.174 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:32:33.174 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:32:33.174 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:32:33.174 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53882/$cluster-$name.asok' 2024-09-16T05:32:33.175 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:32:33.175 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-stop/x '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53882/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:32:33.205 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:247: TEST_0_osd: run_osd td/ok-to-stop 0 2024-09-16T05:32:33.205 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-09-16T05:32:33.205 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:32:33.205 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T05:32:33.205 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:32:33.205 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/0 2024-09-16T05:32:33.206 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=018ac118-e789-4c01-be89-78aa14fbe62f --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-16T05:32:33.206 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:32:33.206 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:32:33.206 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:32:33.206 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/0' 2024-09-16T05:32:33.206 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/0/journal' 2024-09-16T05:32:33.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:32:33.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:32:33.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-09-16T05:32:33.207 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:32:33.207 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:32:33.207 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:32:33.207 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:32:33.207 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:32:33.207 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:32:33.208 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53882/$cluster-$name.asok' 2024-09-16T05:32:33.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53882/$cluster-$name.asok' 2024-09-16T05:32:33.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:32:33.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:32:33.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:32:33.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-09-16T05:32:33.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-09-16T05:32:33.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:32:33.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:32:33.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:32:33.209 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:32:33.209 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:32:33.209 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:32:33.209 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/0 2024-09-16T05:32:33.209 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:32:33.209 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b9f247e9-af8c-432c-b7ee-3097b75c9c54 2024-09-16T05:32:33.209 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b9f247e9-af8c-432c-b7ee-3097b75c9c54' 2024-09-16T05:32:33.210 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 b9f247e9-af8c-432c-b7ee-3097b75c9c54 2024-09-16T05:32:33.210 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:32:33.219 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDxwudmnOAHDRAAZrUyqkz6mZUrKWPEBnBlEw== 2024-09-16T05:32:33.220 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDxwudmnOAHDRAAZrUyqkz6mZUrKWPEBnBlEw=="}' 2024-09-16T05:32:33.220 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b9f247e9-af8c-432c-b7ee-3097b75c9c54 -i td/ok-to-stop/0/new.json 2024-09-16T05:32:33.501 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:32:33.511 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/0/new.json 2024-09-16T05:32:33.512 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=018ac118-e789-4c01-be89-78aa14fbe62f --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53882/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDxwudmnOAHDRAAZrUyqkz6mZUrKWPEBnBlEw== --osd-uuid b9f247e9-af8c-432c-b7ee-3097b75c9c54 2024-09-16T05:32:33.539 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:32:33.538+0000 7f3ffa244640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:32:33.570 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:32:33.569+0000 7f3ffa244640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:32:33.571 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:32:33.570+0000 7f3ffa244640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:32:33.571 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:32:33.570+0000 7f3ffa244640 -1 bdev(0x55f788467c00 td/ok-to-stop/0/block) open stat got: (1) Operation not permitted 2024-09-16T05:32:33.571 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:32:33.570+0000 7f3ffa244640 -1 bluestore(td/ok-to-stop/0) _read_fsid unparsable uuid 2024-09-16T05:32:38.764 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/0/keyring 2024-09-16T05:32:38.764 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:32:38.765 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T05:32:38.765 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T05:32:38.765 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:32:39.276 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T05:32:39.276 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:32:39.277 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=018ac118-e789-4c01-be89-78aa14fbe62f --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53882/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:32:39.277 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:32:39.277 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:32:39.277 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:32:39.300 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:32:39.299+0000 7f5f8fa18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:32:39.313 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:32:39.312+0000 7f5f8fa18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:32:39.314 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:32:39.313+0000 7f5f8fa18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:32:39.671 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T05:32:39.671 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:32:39.671 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:32:39.671 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:32:39.671 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:32:39.671 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:32:39.672 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:32:39.672 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:32:39.672 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:32:39.672 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:32:40.077 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:32:40.671 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:32:40.670+0000 7f5f8fa18640 -1 Falling back to public interface 2024-09-16T05:32:41.078 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:32:41.078 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:32:41.078 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:32:41.079 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:32:41.079 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:32:41.079 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:32:41.486 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:32:42.488 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:32:42.488 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:32:42.488 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:32:42.488 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:32:42.489 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:32:42.489 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:32:42.915 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:32:43.440 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:32:43.439+0000 7f5f8fa18640 -1 osd.0 0 log_to_monitors true 2024-09-16T05:32:43.917 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:32:43.917 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:32:43.917 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:32:43.918 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:32:43.918 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:32:43.918 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:32:44.469 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:32:45.470 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:32:45.470 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:32:45.471 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:32:45.471 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:32:45.471 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:32:45.471 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:32:45.864 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:32:46.865 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:32:46.865 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:32:46.866 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:32:46.866 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:32:46.866 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:32:46.866 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:32:47.314 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:32:48.315 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:32:48.316 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:32:48.316 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:32:48.316 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:32:48.316 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:32:48.317 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:32:48.717 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:32:49.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:32:49.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:32:49.719 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:32:49.720 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:32:49.720 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:32:49.720 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:32:50.166 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:32:51.167 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:32:51.167 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:32:51.167 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:32:51.168 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:32:51.168 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:32:51.168 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:32:51.577 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:32:52.580 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:32:52.580 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:32:52.580 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:32:52.581 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:32:52.581 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:32:52.581 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:32:52.966 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:32:53.967 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:32:53.968 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:32:53.968 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:32:53.969 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:32:53.969 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:32:53.969 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:32:54.705 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:32:55.706 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:32:55.707 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:32:55.707 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:32:55.707 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:32:55.708 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:32:55.708 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:32:56.600 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:32:57.602 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:32:57.602 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:32:57.602 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:32:57.603 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:32:57.603 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:32:57.603 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:32:58.022 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:32:59.023 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:32:59.024 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:32:59.024 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:32:59.024 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:32:59.025 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:32:59.025 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:32:59.461 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:33:00.462 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:33:00.462 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:33:00.463 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:33:00.463 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:33:00.463 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:33:00.463 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:33:01.489 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:33:02.490 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:33:02.491 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:33:02.491 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:33:02.491 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:33:02.492 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:33:02.492 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:33:03.283 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:33:04.284 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:33:04.284 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:33:04.285 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:33:04.285 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:33:04.285 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:33:04.285 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:33:05.060 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:33:06.061 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:33:06.062 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:33:06.062 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:33:06.063 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:33:06.063 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:33:06.063 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:33:06.438 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3881808491,v1:127.0.0.1:6803/3881808491] [v2:127.0.0.1:6804/3881808491,v1:127.0.0.1:6805/3881808491] exists,up b9f247e9-af8c-432c-b7ee-3097b75c9c54 2024-09-16T05:33:06.442 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:33:06.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:33:06.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:33:06.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:248: TEST_0_osd: run_osd td/ok-to-stop 1 2024-09-16T05:33:06.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-09-16T05:33:06.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:33:06.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T05:33:06.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:33:06.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/1 2024-09-16T05:33:06.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=018ac118-e789-4c01-be89-78aa14fbe62f --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-16T05:33:06.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:33:06.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:33:06.444 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:33:06.444 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/1' 2024-09-16T05:33:06.444 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/1/journal' 2024-09-16T05:33:06.444 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:33:06.444 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:33:06.444 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-09-16T05:33:06.444 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:33:06.444 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:33:06.444 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:33:06.444 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:33:06.444 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:33:06.444 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:33:06.444 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53882/$cluster-$name.asok' 2024-09-16T05:33:06.445 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53882/$cluster-$name.asok' 2024-09-16T05:33:06.445 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:33:06.445 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:33:06.445 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:33:06.445 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-09-16T05:33:06.445 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-09-16T05:33:06.445 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:33:06.445 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:33:06.445 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:33:06.445 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:33:06.445 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:33:06.445 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:33:06.446 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/1 2024-09-16T05:33:06.446 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:33:06.446 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f7ec3478-f76a-4d47-961e-aee61e56e1a1 2024-09-16T05:33:06.446 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f7ec3478-f76a-4d47-961e-aee61e56e1a1' 2024-09-16T05:33:06.447 INFO:tasks.workunit.client.0.smithi178.stdout:add osd1 f7ec3478-f76a-4d47-961e-aee61e56e1a1 2024-09-16T05:33:06.448 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:33:06.459 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQASw+dmDzpSGxAAtRm4FAzLyFsLKrjkPmIrxQ== 2024-09-16T05:33:06.459 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQASw+dmDzpSGxAAtRm4FAzLyFsLKrjkPmIrxQ=="}' 2024-09-16T05:33:06.459 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f7ec3478-f76a-4d47-961e-aee61e56e1a1 -i td/ok-to-stop/1/new.json 2024-09-16T05:33:06.885 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:33:06.895 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/1/new.json 2024-09-16T05:33:06.897 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=018ac118-e789-4c01-be89-78aa14fbe62f --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/1 --osd-journal=td/ok-to-stop/1/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53882/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQASw+dmDzpSGxAAtRm4FAzLyFsLKrjkPmIrxQ== --osd-uuid f7ec3478-f76a-4d47-961e-aee61e56e1a1 2024-09-16T05:33:06.924 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:33:06.923+0000 7fe45fb89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:33:06.940 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:33:06.940+0000 7fe45fb89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:33:06.941 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:33:06.941+0000 7fe45fb89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:33:06.942 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:33:06.941+0000 7fe45fb89640 -1 bdev(0x559e6e1a7c00 td/ok-to-stop/1/block) open stat got: (1) Operation not permitted 2024-09-16T05:33:06.942 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:33:06.941+0000 7fe45fb89640 -1 bluestore(td/ok-to-stop/1) _read_fsid unparsable uuid 2024-09-16T05:33:12.584 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/1/keyring 2024-09-16T05:33:12.584 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:33:12.584 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd1 key to auth repository 2024-09-16T05:33:12.585 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T05:33:12.585 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:33:13.102 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.1 2024-09-16T05:33:13.103 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T05:33:13.103 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=018ac118-e789-4c01-be89-78aa14fbe62f --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/1 --osd-journal=td/ok-to-stop/1/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53882/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:33:13.103 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:33:13.103 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:33:13.103 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:33:13.128 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:33:13.127+0000 7feda6c19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:33:13.150 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:33:13.150+0000 7feda6c19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:33:13.151 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:33:13.151+0000 7feda6c19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:33:13.540 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T05:33:13.540 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:33:13.540 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T05:33:13.540 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:33:13.540 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:33:13.541 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:33:13.541 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:33:13.542 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:33:13.542 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:33:13.542 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:33:13.913 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:33:14.498 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:33:14.498+0000 7feda6c19640 -1 Falling back to public interface 2024-09-16T05:33:14.915 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:33:14.915 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:33:14.915 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:33:14.915 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:33:14.916 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:33:14.916 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:33:15.300 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:33:16.301 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:33:16.301 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:33:16.302 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:33:16.302 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:33:16.302 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:33:16.302 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:33:16.717 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:33:17.531 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:33:17.530+0000 7feda6c19640 -1 osd.1 0 log_to_monitors true 2024-09-16T05:33:17.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:33:17.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:33:17.719 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:33:17.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:33:17.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:33:17.720 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:33:18.224 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:33:19.225 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:33:19.225 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:33:19.226 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:33:19.226 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:33:19.226 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:33:19.226 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:33:19.603 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:33:20.605 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:33:20.605 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:33:20.605 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:33:20.606 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:33:20.606 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:33:20.606 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:33:21.010 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:33:22.012 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:33:22.012 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:33:22.012 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:33:22.013 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:33:22.013 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:33:22.013 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:33:22.483 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:33:23.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:33:23.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:33:23.486 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:33:23.486 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:33:23.486 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:33:23.486 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:33:23.886 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:33:24.888 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:33:24.888 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:33:24.889 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:33:24.890 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:33:24.890 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:33:24.890 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:33:25.274 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:33:26.276 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:33:26.276 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:33:26.277 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:33:26.277 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:33:26.277 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:33:26.277 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:33:26.652 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:33:27.653 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:33:27.656 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:33:27.656 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:33:27.656 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:33:27.657 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:33:27.657 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:33:28.065 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:33:29.066 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:33:29.067 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:33:29.067 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:33:29.067 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:33:29.067 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:33:29.068 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:33:29.713 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:33:30.714 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:33:30.714 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:33:30.715 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:33:30.715 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:33:30.715 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:33:30.715 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:33:31.319 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:33:32.320 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:33:32.321 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:33:32.321 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:33:32.321 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:33:32.321 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:33:32.321 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:33:33.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:33:34.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:33:34.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:33:34.044 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:33:34.045 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:33:34.045 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:33:34.046 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:33:34.787 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:33:35.788 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:33:35.788 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:33:35.789 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:33:35.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:33:35.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:33:35.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:33:36.891 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:33:37.893 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:33:37.893 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:33:37.893 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:33:37.894 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:33:37.894 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:33:37.894 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:33:38.736 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:33:39.740 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:33:39.741 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:33:39.741 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:33:39.741 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:33:39.742 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:33:39.742 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:33:40.174 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:33:41.175 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:33:41.176 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:33:41.176 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T05:33:41.176 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T05:33:41.176 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:33:41.177 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:33:41.643 INFO:tasks.workunit.client.0.smithi178.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3635401712,v1:127.0.0.1:6811/3635401712] [v2:127.0.0.1:6812/3635401712,v1:127.0.0.1:6813/3635401712] exists,up f7ec3478-f76a-4d47-961e-aee61e56e1a1 2024-09-16T05:33:41.647 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:33:41.648 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:33:41.648 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:33:41.648 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:249: TEST_0_osd: run_osd td/ok-to-stop 2 2024-09-16T05:33:41.648 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-09-16T05:33:41.648 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:33:41.648 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T05:33:41.648 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:33:41.648 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/2 2024-09-16T05:33:41.648 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=018ac118-e789-4c01-be89-78aa14fbe62f --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-16T05:33:41.648 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:33:41.648 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:33:41.648 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:33:41.649 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/2' 2024-09-16T05:33:41.649 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/2/journal' 2024-09-16T05:33:41.649 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:33:41.649 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:33:41.649 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-09-16T05:33:41.649 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:33:41.649 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:33:41.649 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:33:41.649 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:33:41.649 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:33:41.649 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:33:41.649 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53882/$cluster-$name.asok' 2024-09-16T05:33:41.649 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53882/$cluster-$name.asok' 2024-09-16T05:33:41.649 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:33:41.650 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:33:41.650 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:33:41.650 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-09-16T05:33:41.650 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-09-16T05:33:41.650 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:33:41.650 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:33:41.650 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:33:41.650 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:33:41.650 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:33:41.650 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:33:41.650 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/2 2024-09-16T05:33:41.650 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:33:41.650 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=327671c4-eb7c-4277-9286-f15f2d34297f 2024-09-16T05:33:41.651 INFO:tasks.workunit.client.0.smithi178.stdout:add osd2 327671c4-eb7c-4277-9286-f15f2d34297f 2024-09-16T05:33:41.651 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 327671c4-eb7c-4277-9286-f15f2d34297f' 2024-09-16T05:33:41.651 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:33:41.664 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA1w+dmzFmJJxAAYO2shdtOA+gGdNKnAyxokw== 2024-09-16T05:33:41.664 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA1w+dmzFmJJxAAYO2shdtOA+gGdNKnAyxokw=="}' 2024-09-16T05:33:41.664 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 327671c4-eb7c-4277-9286-f15f2d34297f -i td/ok-to-stop/2/new.json 2024-09-16T05:33:42.107 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:33:42.118 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/2/new.json 2024-09-16T05:33:42.119 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=018ac118-e789-4c01-be89-78aa14fbe62f --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/2 --osd-journal=td/ok-to-stop/2/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53882/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA1w+dmzFmJJxAAYO2shdtOA+gGdNKnAyxokw== --osd-uuid 327671c4-eb7c-4277-9286-f15f2d34297f 2024-09-16T05:33:42.145 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:33:42.145+0000 7fd3bdd45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:33:42.159 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:33:42.159+0000 7fd3bdd45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:33:42.160 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:33:42.160+0000 7fd3bdd45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:33:42.161 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:33:42.160+0000 7fd3bdd45640 -1 bdev(0x55f62f6ffc00 td/ok-to-stop/2/block) open stat got: (1) Operation not permitted 2024-09-16T05:33:42.161 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:33:42.160+0000 7fd3bdd45640 -1 bluestore(td/ok-to-stop/2) _read_fsid unparsable uuid 2024-09-16T05:33:48.445 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/2/keyring 2024-09-16T05:33:48.446 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:33:48.446 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd2 key to auth repository 2024-09-16T05:33:48.447 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T05:33:48.447 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:33:48.956 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.2 2024-09-16T05:33:48.956 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T05:33:48.957 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=018ac118-e789-4c01-be89-78aa14fbe62f --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/2 --osd-journal=td/ok-to-stop/2/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53882/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:33:48.957 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:33:48.957 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:33:48.957 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:33:48.981 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:33:48.979+0000 7ff2c4c2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:33:49.004 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:33:49.003+0000 7ff2c4c2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:33:49.005 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:33:49.004+0000 7ff2c4c2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:33:49.421 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T05:33:49.421 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:33:49.421 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T05:33:49.421 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:33:49.422 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:33:49.422 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:33:49.422 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:33:49.422 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:33:49.422 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:33:49.422 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:33:49.791 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:33:50.111 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:33:50.110+0000 7ff2c4c2e640 -1 Falling back to public interface 2024-09-16T05:33:50.792 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:33:50.792 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:33:50.792 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:33:50.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:33:50.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:33:50.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:33:51.229 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:33:52.232 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:33:52.233 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:33:52.233 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:33:52.233 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:33:52.234 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:33:52.234 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:33:52.647 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:33:53.309 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:33:53.308+0000 7ff2c4c2e640 -1 osd.2 0 log_to_monitors true 2024-09-16T05:33:53.648 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:33:53.648 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:33:53.648 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:33:53.649 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:33:53.649 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:33:53.649 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:33:54.194 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:33:55.196 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:33:55.196 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:33:55.196 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:33:55.197 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:33:55.197 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:33:55.197 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:33:55.634 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:33:56.635 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:33:56.636 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:33:56.636 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:33:56.636 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:33:56.636 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:33:56.637 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:33:57.025 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:33:58.027 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:33:58.027 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:33:58.027 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:33:58.028 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:33:58.028 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:33:58.028 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:33:58.462 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:33:59.463 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:33:59.464 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:33:59.464 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:33:59.465 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:33:59.465 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:33:59.465 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:33:59.960 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:34:00.962 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:34:00.962 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:34:00.962 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:34:00.963 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:34:00.963 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:34:00.963 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:34:01.450 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:34:02.451 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:34:02.451 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:34:02.452 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:34:02.452 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:34:02.452 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:34:02.452 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:34:02.856 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:34:03.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:34:03.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:34:03.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:34:03.859 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:34:03.859 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:34:03.859 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:34:04.361 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:34:05.362 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:34:05.363 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:34:05.363 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:34:05.363 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:34:05.363 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:34:05.363 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:34:05.841 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:34:06.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:34:06.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:34:06.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:34:06.843 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:34:06.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:34:06.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:34:07.267 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:34:08.269 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:34:08.269 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:34:08.269 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:34:08.269 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:34:08.270 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:34:08.270 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:34:09.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:34:10.045 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:34:10.045 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:34:10.046 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:34:10.046 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:34:10.046 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:34:10.046 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:34:10.738 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:34:11.740 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:34:11.740 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:34:11.740 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:34:11.741 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:34:11.741 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:34:11.741 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:34:12.921 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:34:13.922 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:34:13.922 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:34:13.923 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:34:13.923 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:34:13.923 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:34:13.923 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:34:14.702 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:34:15.704 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:34:15.704 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:34:15.704 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:34:15.705 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:34:15.705 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:34:15.705 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:34:16.414 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:34:17.415 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:34:17.416 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:34:17.416 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T05:34:17.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T05:34:17.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:34:17.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:34:17.796 INFO:tasks.workunit.client.0.smithi178.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3008610263,v1:127.0.0.1:6819/3008610263] [v2:127.0.0.1:6820/3008610263,v1:127.0.0.1:6821/3008610263] exists,up 327671c4-eb7c-4277-9286-f15f2d34297f 2024-09-16T05:34:17.800 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:34:17.800 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:34:17.801 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:34:17.801 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:250: TEST_0_osd: run_osd td/ok-to-stop 3 2024-09-16T05:34:17.801 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-09-16T05:34:17.801 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:34:17.801 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-16T05:34:17.801 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:34:17.801 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/3 2024-09-16T05:34:17.801 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=018ac118-e789-4c01-be89-78aa14fbe62f --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-16T05:34:17.801 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:34:17.801 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:34:17.801 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:34:17.801 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/3' 2024-09-16T05:34:17.801 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/3/journal' 2024-09-16T05:34:17.802 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:34:17.802 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:34:17.802 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-09-16T05:34:17.802 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:34:17.802 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:34:17.802 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:34:17.802 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:34:17.802 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:34:17.802 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:34:17.802 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53882/$cluster-$name.asok' 2024-09-16T05:34:17.802 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53882/$cluster-$name.asok' 2024-09-16T05:34:17.802 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:34:17.802 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:34:17.803 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:34:17.803 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-09-16T05:34:17.803 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-09-16T05:34:17.803 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:34:17.803 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:34:17.803 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:34:17.803 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:34:17.803 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:34:17.803 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:34:17.803 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/3 2024-09-16T05:34:17.803 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:34:17.803 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3c356513-8bf3-43b4-9f62-601481a65b93 2024-09-16T05:34:17.804 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 3c356513-8bf3-43b4-9f62-601481a65b93' 2024-09-16T05:34:17.804 INFO:tasks.workunit.client.0.smithi178.stdout:add osd3 3c356513-8bf3-43b4-9f62-601481a65b93 2024-09-16T05:34:17.804 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:34:17.817 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBZw+dmKDmmMBAA7j+iaAaALR8KnqfT35pbmA== 2024-09-16T05:34:17.817 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBZw+dmKDmmMBAA7j+iaAaALR8KnqfT35pbmA=="}' 2024-09-16T05:34:17.817 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3c356513-8bf3-43b4-9f62-601481a65b93 -i td/ok-to-stop/3/new.json 2024-09-16T05:34:18.401 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:34:18.413 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/3/new.json 2024-09-16T05:34:18.413 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=018ac118-e789-4c01-be89-78aa14fbe62f --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/3 --osd-journal=td/ok-to-stop/3/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53882/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBZw+dmKDmmMBAA7j+iaAaALR8KnqfT35pbmA== --osd-uuid 3c356513-8bf3-43b4-9f62-601481a65b93 2024-09-16T05:34:18.440 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:34:18.439+0000 7f23a2e33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:34:18.460 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:34:18.459+0000 7f23a2e33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:34:18.460 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:34:18.460+0000 7f23a2e33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:34:18.461 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:34:18.460+0000 7f23a2e33640 -1 bdev(0x560fdb6adc00 td/ok-to-stop/3/block) open stat got: (1) Operation not permitted 2024-09-16T05:34:18.462 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:34:18.460+0000 7f23a2e33640 -1 bluestore(td/ok-to-stop/3) _read_fsid unparsable uuid 2024-09-16T05:34:24.292 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/3/keyring 2024-09-16T05:34:24.292 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:34:24.293 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd3 key to auth repository 2024-09-16T05:34:24.294 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-16T05:34:24.294 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:34:24.803 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.3 2024-09-16T05:34:24.804 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-16T05:34:24.804 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=018ac118-e789-4c01-be89-78aa14fbe62f --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/3 --osd-journal=td/ok-to-stop/3/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53882/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:34:24.804 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:34:24.804 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:34:24.804 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:34:24.828 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:34:24.827+0000 7f987194d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:34:24.830 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:34:24.829+0000 7f987194d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:34:24.831 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:34:24.830+0000 7f987194d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:34:25.240 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-16T05:34:25.240 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:34:25.240 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T05:34:25.240 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:34:25.241 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:34:25.241 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:34:25.241 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:34:25.241 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:34:25.241 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:34:25.241 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T05:34:25.648 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:34:26.650 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:34:26.650 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:34:26.651 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:34:26.651 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:34:26.651 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:34:26.651 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T05:34:26.699 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:34:26.698+0000 7f987194d640 -1 Falling back to public interface 2024-09-16T05:34:27.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:34:28.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:34:28.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:34:28.031 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:34:28.032 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:34:28.032 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:34:28.032 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T05:34:28.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:34:29.445 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:34:29.445 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:34:29.446 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:34:29.446 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:34:29.446 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:34:29.446 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T05:34:29.482 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:34:29.482+0000 7f987194d640 -1 osd.3 0 log_to_monitors true 2024-09-16T05:34:29.967 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:34:30.968 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:34:30.968 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:34:30.969 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:34:30.969 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:34:30.969 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:34:30.969 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T05:34:31.473 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:34:32.474 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:34:32.475 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:34:32.475 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:34:32.475 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:34:32.475 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:34:32.476 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T05:34:32.860 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:34:33.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:34:33.862 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:34:33.862 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:34:33.862 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:34:33.863 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:34:33.863 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T05:34:34.304 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:34:35.305 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:34:35.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:34:35.306 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:34:35.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:34:35.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:34:35.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T05:34:35.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:34:36.720 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:34:36.721 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:34:36.721 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:34:36.721 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:34:36.722 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:34:36.722 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T05:34:37.098 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:34:38.101 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:34:38.101 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:34:38.101 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:34:38.101 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:34:38.102 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:34:38.102 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T05:34:38.598 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:34:39.599 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:34:39.600 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:34:39.600 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:34:39.600 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:34:39.601 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:34:39.601 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T05:34:40.073 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:34:41.074 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:34:41.074 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:34:41.074 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:34:41.075 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:34:41.075 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:34:41.075 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T05:34:42.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:34:43.002 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:34:43.002 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:34:43.002 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:34:43.003 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:34:43.003 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:34:43.003 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T05:34:43.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:34:44.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:34:44.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:34:44.486 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:34:44.486 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:34:44.486 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:34:44.486 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T05:34:45.527 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:34:46.528 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:34:46.528 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:34:46.528 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:34:46.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:34:46.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:34:46.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T05:34:47.362 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:34:48.364 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:34:48.364 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:34:48.364 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:34:48.365 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:34:48.365 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:34:48.365 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T05:34:49.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:34:50.209 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:34:50.209 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:34:50.210 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:34:50.210 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:34:50.210 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:34:50.210 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T05:34:50.797 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:34:51.799 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:34:51.799 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:34:51.799 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:34:51.799 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:34:51.800 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:34:51.800 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T05:34:52.460 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:34:53.462 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:34:53.462 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:34:53.462 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T05:34:53.463 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T05:34:53.463 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:34:53.463 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T05:34:53.896 INFO:tasks.workunit.client.0.smithi178.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1262161685,v1:127.0.0.1:6827/1262161685] [v2:127.0.0.1:6828/1262161685,v1:127.0.0.1:6829/1262161685] exists,up 3c356513-8bf3-43b4-9f62-601481a65b93 2024-09-16T05:34:53.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:34:53.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:34:53.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:34:53.897 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:252: TEST_0_osd: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2024-09-16T05:34:54.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:253: TEST_0_osd: ceph osd pool create ec erasure ec-profile 2024-09-16T05:34:55.202 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'ec' already exists 2024-09-16T05:34:55.214 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:255: TEST_0_osd: wait_for_clean 2024-09-16T05:34:55.214 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T05:34:55.215 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T05:34:55.215 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T05:34:55.215 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T05:34:55.215 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T05:34:55.215 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T05:34:55.215 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T05:34:55.215 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T05:34:55.215 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T05:34:55.300 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T05:34:55.300 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T05:34:55.300 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T05:34:55.300 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T05:34:55.300 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-16T05:34:55.301 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-16T05:34:55.941 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-16T05:34:55.941 INFO:tasks.workunit.client.0.smithi178.stderr:1 2024-09-16T05:34:55.941 INFO:tasks.workunit.client.0.smithi178.stderr:2 2024-09-16T05:34:55.941 INFO:tasks.workunit.client.0.smithi178.stderr:3' 2024-09-16T05:34:55.941 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-16T05:34:55.942 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:34:55.942 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T05:34:56.195 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836509 2024-09-16T05:34:56.195 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836509 2024-09-16T05:34:56.195 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509' 2024-09-16T05:34:56.195 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:34:56.196 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T05:34:56.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672983 2024-09-16T05:34:56.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672983 2024-09-16T05:34:56.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672983' 2024-09-16T05:34:56.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:34:56.401 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T05:34:56.629 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509457 2024-09-16T05:34:56.630 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509457 2024-09-16T05:34:56.630 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672983 2-64424509457' 2024-09-16T05:34:56.630 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:34:56.630 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T05:34:56.747 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345926 2024-09-16T05:34:56.748 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345926 2024-09-16T05:34:56.748 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672983 2-64424509457 3-85899345926' 2024-09-16T05:34:56.748 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:34:56.748 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836509 2024-09-16T05:34:56.748 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:34:56.749 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-16T05:34:56.749 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836509 2024-09-16T05:34:56.749 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:34:56.750 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836509 2024-09-16T05:34:56.750 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-09-16T05:34:56.751 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.0 seq 21474836509 2024-09-16T05:34:56.751 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:34:57.367 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836509 -lt 21474836509 2024-09-16T05:34:57.367 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:34:57.367 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672983 2024-09-16T05:34:57.367 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:34:57.368 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-09-16T05:34:57.369 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672983 2024-09-16T05:34:57.369 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:34:57.370 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672983 2024-09-16T05:34:57.370 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.1 seq 42949672983 2024-09-16T05:34:57.371 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-09-16T05:34:57.371 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T05:34:57.740 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672983 -lt 42949672983 2024-09-16T05:34:57.740 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:34:57.741 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509457 2024-09-16T05:34:57.741 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:34:57.741 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-09-16T05:34:57.742 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509457 2024-09-16T05:34:57.742 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:34:57.743 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509457 2024-09-16T05:34:57.743 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.2 seq 64424509457 2024-09-16T05:34:57.744 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-09-16T05:34:57.744 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T05:34:58.145 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509457 -lt 64424509457 2024-09-16T05:34:58.145 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:34:58.146 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345926 2024-09-16T05:34:58.146 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:34:58.146 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-09-16T05:34:58.147 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345926 2024-09-16T05:34:58.147 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:34:58.148 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345926 2024-09-16T05:34:58.148 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.3 seq 85899345926 2024-09-16T05:34:58.148 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345926' 2024-09-16T05:34:58.148 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T05:34:58.577 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345926 -lt 85899345926 2024-09-16T05:34:58.578 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T05:34:58.578 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:34:58.578 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:34:59.077 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-09-16T05:34:59.077 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T05:34:59.077 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T05:34:59.078 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T05:34:59.078 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T05:34:59.078 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T05:34:59.078 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T05:34:59.078 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T05:34:59.505 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=11 2024-09-16T05:34:59.506 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T05:34:59.506 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:34:59.506 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:35:00.001 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 11 = 32 2024-09-16T05:35:00.001 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 11 '!=' -1 2024-09-16T05:35:00.001 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-16T05:35:00.001 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=11 2024-09-16T05:35:00.001 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T05:35:00.001 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T05:35:00.103 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T05:35:00.103 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T05:35:00.103 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T05:35:00.103 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T05:35:00.104 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T05:35:00.104 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T05:35:00.104 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T05:35:00.104 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T05:35:00.469 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=11 2024-09-16T05:35:00.470 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T05:35:00.470 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:35:00.470 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:35:00.966 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 11 = 32 2024-09-16T05:35:00.966 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 11 '!=' 11 2024-09-16T05:35:00.966 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T05:35:00.967 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T05:35:00.967 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T05:35:00.967 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T05:35:00.967 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T05:35:00.967 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T05:35:00.967 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T05:35:01.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T05:35:01.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T05:35:01.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-09-16T05:35:01.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T05:35:01.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-16T05:35:01.731 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T05:35:01.732 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T05:35:01.732 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T05:35:01.732 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T05:35:01.732 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T05:35:01.732 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T05:35:01.732 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T05:35:01.732 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T05:35:02.095 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=19 2024-09-16T05:35:02.095 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T05:35:02.095 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:35:02.095 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:35:02.589 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 19 = 32 2024-09-16T05:35:02.590 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 19 '!=' 11 2024-09-16T05:35:02.590 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-16T05:35:02.590 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=19 2024-09-16T05:35:02.590 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T05:35:02.590 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T05:35:02.691 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T05:35:02.691 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T05:35:02.691 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T05:35:02.691 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T05:35:02.692 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T05:35:02.692 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T05:35:02.692 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T05:35:02.692 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T05:35:03.059 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-09-16T05:35:03.059 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T05:35:03.059 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:35:03.060 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:35:03.560 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-09-16T05:35:03.560 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T05:35:03.560 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T05:35:03.560 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:258: TEST_0_osd: ceph osd pool set ec min_size 3 2024-09-16T05:35:04.197 INFO:tasks.workunit.client.0.smithi178.stderr:set pool 1 min_size to 3 2024-09-16T05:35:04.209 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:259: TEST_0_osd: wait_for_clean 2024-09-16T05:35:04.209 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T05:35:04.209 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T05:35:04.209 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T05:35:04.209 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T05:35:04.210 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T05:35:04.210 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T05:35:04.210 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T05:35:04.210 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T05:35:04.210 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T05:35:04.296 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T05:35:04.296 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T05:35:04.296 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T05:35:04.296 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T05:35:04.296 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-16T05:35:04.296 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-16T05:35:04.760 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-16T05:35:04.761 INFO:tasks.workunit.client.0.smithi178.stderr:1 2024-09-16T05:35:04.761 INFO:tasks.workunit.client.0.smithi178.stderr:2 2024-09-16T05:35:04.761 INFO:tasks.workunit.client.0.smithi178.stderr:3' 2024-09-16T05:35:04.761 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-16T05:35:04.761 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:35:04.761 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T05:35:04.872 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836512 2024-09-16T05:35:04.872 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836512 2024-09-16T05:35:04.873 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512' 2024-09-16T05:35:04.873 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:35:04.873 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T05:35:05.005 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672986 2024-09-16T05:35:05.005 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672986 2024-09-16T05:35:05.005 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512 1-42949672986' 2024-09-16T05:35:05.005 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:35:05.005 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T05:35:05.114 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509461 2024-09-16T05:35:05.114 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509461 2024-09-16T05:35:05.115 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512 1-42949672986 2-64424509461' 2024-09-16T05:35:05.115 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:35:05.115 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T05:35:05.225 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345930 2024-09-16T05:35:05.225 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345930 2024-09-16T05:35:05.225 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512 1-42949672986 2-64424509461 3-85899345930' 2024-09-16T05:35:05.225 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:35:05.225 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836512 2024-09-16T05:35:05.226 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:35:05.226 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-16T05:35:05.226 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836512 2024-09-16T05:35:05.226 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:35:05.227 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836512 2024-09-16T05:35:05.227 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.0 seq 21474836512 2024-09-16T05:35:05.228 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-09-16T05:35:05.228 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:35:05.641 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836512 -lt 21474836512 2024-09-16T05:35:05.641 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:35:05.642 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672986 2024-09-16T05:35:05.642 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:35:05.643 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-09-16T05:35:05.643 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672986 2024-09-16T05:35:05.643 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:35:05.644 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672986 2024-09-16T05:35:05.644 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.1 seq 42949672986 2024-09-16T05:35:05.645 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2024-09-16T05:35:05.645 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T05:35:06.011 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672986 -lt 42949672986 2024-09-16T05:35:06.011 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:35:06.012 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509461 2024-09-16T05:35:06.012 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:35:06.012 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-09-16T05:35:06.013 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509461 2024-09-16T05:35:06.013 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:35:06.014 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509461 2024-09-16T05:35:06.014 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.2 seq 64424509461 2024-09-16T05:35:06.014 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-09-16T05:35:06.014 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T05:35:06.382 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509461 2024-09-16T05:35:06.382 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-16T05:35:07.384 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T05:35:07.384 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T05:35:07.813 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509461 -lt 64424509461 2024-09-16T05:35:07.813 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:35:07.813 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345930 2024-09-16T05:35:07.813 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:35:07.814 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-09-16T05:35:07.814 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345930 2024-09-16T05:35:07.814 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:35:07.815 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345930 2024-09-16T05:35:07.815 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.3 seq 85899345930 2024-09-16T05:35:07.816 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345930' 2024-09-16T05:35:07.816 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T05:35:08.264 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345930 -lt 85899345930 2024-09-16T05:35:08.264 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T05:35:08.265 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:35:08.265 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:35:08.765 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-09-16T05:35:08.765 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T05:35:08.765 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T05:35:08.765 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T05:35:08.765 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T05:35:08.765 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T05:35:08.765 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T05:35:08.765 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T05:35:09.166 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-09-16T05:35:09.166 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T05:35:09.167 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:35:09.167 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:35:09.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-09-16T05:35:09.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T05:35:09.663 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T05:35:09.663 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:261: TEST_0_osd: ceph osd ok-to-stop 0 2024-09-16T05:35:10.014 INFO:tasks.workunit.client.0.smithi178.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T05:35:10.025 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:262: TEST_0_osd: ceph osd ok-to-stop 1 2024-09-16T05:35:10.381 INFO:tasks.workunit.client.0.smithi178.stdout:{"ok_to_stop":true,"osds":[1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T05:35:10.392 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:263: TEST_0_osd: ceph osd ok-to-stop 2 2024-09-16T05:35:10.745 INFO:tasks.workunit.client.0.smithi178.stdout:{"ok_to_stop":true,"osds":[2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T05:35:10.756 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:264: TEST_0_osd: ceph osd ok-to-stop 3 2024-09-16T05:35:11.110 INFO:tasks.workunit.client.0.smithi178.stdout:{"ok_to_stop":true,"osds":[3],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T05:35:11.121 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:265: TEST_0_osd: ceph osd ok-to-stop 0 1 2024-09-16T05:35:11.375 INFO:tasks.workunit.client.0.smithi178.stdout:{"ok_to_stop":false,"osds":[0,1],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T05:35:11.375 INFO:tasks.workunit.client.0.smithi178.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-09-16T05:35:11.379 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:266: TEST_0_osd: ceph osd ok-to-stop 2 3 2024-09-16T05:35:11.632 INFO:tasks.workunit.client.0.smithi178.stdout:{"ok_to_stop":false,"osds":[2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T05:35:11.632 INFO:tasks.workunit.client.0.smithi178.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-09-16T05:35:11.635 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:267: TEST_0_osd: ceph osd ok-to-stop 0 --max 2 2024-09-16T05:35:11.636 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:267: TEST_0_osd: grep '[0]' 2024-09-16T05:35:12.004 INFO:tasks.workunit.client.0.smithi178.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T05:35:12.005 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:268: TEST_0_osd: ceph osd ok-to-stop 1 --max 2 2024-09-16T05:35:12.005 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:268: TEST_0_osd: grep '[1]' 2024-09-16T05:35:12.368 INFO:tasks.workunit.client.0.smithi178.stdout:{"ok_to_stop":true,"osds":[1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T05:35:12.369 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:271: TEST_0_osd: ceph osd pool set ec min_size 2 2024-09-16T05:35:13.143 INFO:tasks.workunit.client.0.smithi178.stderr:set pool 1 min_size to 2 2024-09-16T05:35:13.157 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:272: TEST_0_osd: wait_for_clean 2024-09-16T05:35:13.157 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T05:35:13.157 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T05:35:13.157 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T05:35:13.158 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T05:35:13.158 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T05:35:13.158 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T05:35:13.158 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T05:35:13.158 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T05:35:13.158 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T05:35:13.243 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T05:35:13.244 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T05:35:13.244 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T05:35:13.244 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T05:35:13.244 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-16T05:35:13.244 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-16T05:35:13.792 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-16T05:35:13.805 INFO:tasks.workunit.client.0.smithi178.stderr:1 2024-09-16T05:35:13.805 INFO:tasks.workunit.client.0.smithi178.stderr:2 2024-09-16T05:35:13.805 INFO:tasks.workunit.client.0.smithi178.stderr:3' 2024-09-16T05:35:13.805 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-16T05:35:13.805 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:35:13.805 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T05:35:14.063 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836516 2024-09-16T05:35:14.063 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836516 2024-09-16T05:35:14.063 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516' 2024-09-16T05:35:14.063 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:35:14.063 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T05:35:14.178 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672990 2024-09-16T05:35:14.178 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672990 2024-09-16T05:35:14.178 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672990' 2024-09-16T05:35:14.178 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:35:14.179 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T05:35:14.289 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509465 2024-09-16T05:35:14.289 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509465 2024-09-16T05:35:14.289 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672990 2-64424509465' 2024-09-16T05:35:14.289 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:35:14.289 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T05:35:14.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345934 2024-09-16T05:35:14.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345934 2024-09-16T05:35:14.402 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672990 2-64424509465 3-85899345934' 2024-09-16T05:35:14.402 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:35:14.402 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836516 2024-09-16T05:35:14.402 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:35:14.402 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-16T05:35:14.403 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836516 2024-09-16T05:35:14.403 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:35:14.404 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836516 2024-09-16T05:35:14.404 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2024-09-16T05:35:14.404 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.0 seq 21474836516 2024-09-16T05:35:14.404 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:35:14.778 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836514 -lt 21474836516 2024-09-16T05:35:14.778 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-16T05:35:15.779 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T05:35:15.779 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:35:16.186 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836516 -lt 21474836516 2024-09-16T05:35:16.187 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:35:16.187 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672990 2024-09-16T05:35:16.187 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:35:16.188 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-09-16T05:35:16.188 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672990 2024-09-16T05:35:16.188 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:35:16.189 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672990 2024-09-16T05:35:16.189 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.1 seq 42949672990 2024-09-16T05:35:16.190 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672990' 2024-09-16T05:35:16.190 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T05:35:16.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672990 -lt 42949672990 2024-09-16T05:35:16.618 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:35:16.618 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509465 2024-09-16T05:35:16.618 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:35:16.619 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-09-16T05:35:16.619 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509465 2024-09-16T05:35:16.619 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:35:16.620 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509465 2024-09-16T05:35:16.620 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.2 seq 64424509465 2024-09-16T05:35:16.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509465' 2024-09-16T05:35:16.621 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T05:35:16.990 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509465 -lt 64424509465 2024-09-16T05:35:16.990 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:35:16.990 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345934 2024-09-16T05:35:16.990 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:35:16.990 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-09-16T05:35:16.991 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345934 2024-09-16T05:35:16.991 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:35:16.992 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345934 2024-09-16T05:35:16.992 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345934' 2024-09-16T05:35:16.992 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.3 seq 85899345934 2024-09-16T05:35:16.993 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T05:35:17.391 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345934 -lt 85899345934 2024-09-16T05:35:17.391 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T05:35:17.391 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:35:17.391 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:35:17.890 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-09-16T05:35:17.890 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T05:35:17.890 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T05:35:17.890 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T05:35:17.890 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T05:35:17.891 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T05:35:17.891 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T05:35:17.891 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T05:35:18.295 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=25 2024-09-16T05:35:18.295 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T05:35:18.295 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:35:18.295 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:35:18.791 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 25 = 32 2024-09-16T05:35:18.792 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 25 '!=' -1 2024-09-16T05:35:18.792 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-16T05:35:18.792 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=25 2024-09-16T05:35:18.792 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T05:35:18.792 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T05:35:18.893 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T05:35:18.893 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T05:35:18.893 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T05:35:18.893 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T05:35:18.893 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T05:35:18.894 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T05:35:18.894 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T05:35:18.894 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T05:35:19.255 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-09-16T05:35:19.255 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T05:35:19.255 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:35:19.255 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:35:19.792 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-09-16T05:35:19.792 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T05:35:19.792 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T05:35:19.792 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:274: TEST_0_osd: ceph osd ok-to-stop 0 1 2024-09-16T05:35:20.231 INFO:tasks.workunit.client.0.smithi178.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T05:35:20.242 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:275: TEST_0_osd: ceph osd ok-to-stop 2 3 2024-09-16T05:35:20.598 INFO:tasks.workunit.client.0.smithi178.stdout:{"ok_to_stop":true,"osds":[2,3],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T05:35:20.609 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:276: TEST_0_osd: ceph osd ok-to-stop 0 1 2 2024-09-16T05:35:20.863 INFO:tasks.workunit.client.0.smithi178.stdout:{"ok_to_stop":false,"osds":[0,1,2],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T05:35:20.863 INFO:tasks.workunit.client.0.smithi178.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-09-16T05:35:20.867 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:277: TEST_0_osd: ceph osd ok-to-stop 1 2 3 2024-09-16T05:35:21.123 INFO:tasks.workunit.client.0.smithi178.stdout:{"ok_to_stop":false,"osds":[1,2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T05:35:21.124 INFO:tasks.workunit.client.0.smithi178.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-09-16T05:35:21.128 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:279: TEST_0_osd: ceph osd ok-to-stop 0 --max 2 2024-09-16T05:35:21.128 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:279: TEST_0_osd: grep '[0,1]' 2024-09-16T05:35:21.494 INFO:tasks.workunit.client.0.smithi178.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T05:35:21.494 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:280: TEST_0_osd: ceph osd ok-to-stop 0 --max 20 2024-09-16T05:35:21.494 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:280: TEST_0_osd: grep '[0,1]' 2024-09-16T05:35:21.858 INFO:tasks.workunit.client.0.smithi178.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T05:35:21.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:281: TEST_0_osd: ceph osd ok-to-stop 2 --max 2 2024-09-16T05:35:21.859 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:281: TEST_0_osd: grep '[2,3]' 2024-09-16T05:35:22.223 INFO:tasks.workunit.client.0.smithi178.stdout:{"ok_to_stop":true,"osds":[0,2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T05:35:22.224 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:282: TEST_0_osd: ceph osd ok-to-stop 2 --max 20 2024-09-16T05:35:22.224 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:282: TEST_0_osd: grep '[2,3]' 2024-09-16T05:35:22.599 INFO:tasks.workunit.client.0.smithi178.stdout:{"ok_to_stop":true,"osds":[0,2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T05:35:22.600 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:285: TEST_0_osd: kill_daemons td/ok-to-stop TERM osd.0 2024-09-16T05:35:22.600 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:35:22.600 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:35:22.600 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:35:22.600 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:35:22.600 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:35:22.906 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:35:22.906 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:286: TEST_0_osd: ceph osd down 0 2024-09-16T05:35:23.698 INFO:tasks.workunit.client.0.smithi178.stderr:osd.0 is already down. 2024-09-16T05:35:23.709 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:287: TEST_0_osd: wait_for_peered 2024-09-16T05:35:23.709 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: wait_for_peered: local cmd= 2024-09-16T05:35:23.710 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: wait_for_peered: local num_peered=-1 2024-09-16T05:35:23.710 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1701: wait_for_peered: local cur_peered 2024-09-16T05:35:23.710 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: get_timeout_delays 90 .1 2024-09-16T05:35:23.710 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T05:35:23.710 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T05:35:23.711 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T05:35:23.711 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T05:35:23.711 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T05:35:23.798 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T05:35:23.798 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: local -a delays 2024-09-16T05:35:23.798 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_peered: local -i loop=0 2024-09-16T05:35:23.798 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1705: wait_for_peered: flush_pg_stats 2024-09-16T05:35:23.798 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-16T05:35:23.798 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-16T05:35:24.169 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-16T05:35:24.169 INFO:tasks.workunit.client.0.smithi178.stderr:1 2024-09-16T05:35:24.169 INFO:tasks.workunit.client.0.smithi178.stderr:2 2024-09-16T05:35:24.169 INFO:tasks.workunit.client.0.smithi178.stderr:3' 2024-09-16T05:35:24.169 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-16T05:35:24.169 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:35:24.169 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T05:35:24.246 INFO:tasks.workunit.client.0.smithi178.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-16T05:35:24.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq= 2024-09-16T05:35:24.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z '' 2024-09-16T05:35:24.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: continue 2024-09-16T05:35:24.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:35:24.248 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T05:35:24.410 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672994 2024-09-16T05:35:24.410 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672994 2024-09-16T05:35:24.410 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 1-42949672994' 2024-09-16T05:35:24.410 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:35:24.410 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T05:35:24.522 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509469 2024-09-16T05:35:24.522 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509469 2024-09-16T05:35:24.522 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 1-42949672994 2-64424509469' 2024-09-16T05:35:24.522 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:35:24.522 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T05:35:24.637 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345938 2024-09-16T05:35:24.637 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345938 2024-09-16T05:35:24.638 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 1-42949672994 2-64424509469 3-85899345938' 2024-09-16T05:35:24.638 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:35:24.638 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672994 2024-09-16T05:35:24.638 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:35:24.638 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-09-16T05:35:24.638 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672994 2024-09-16T05:35:24.638 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:35:24.638 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672994 2024-09-16T05:35:24.638 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.1 seq 42949672994 2024-09-16T05:35:24.639 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672994' 2024-09-16T05:35:24.639 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T05:35:25.004 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672992 -lt 42949672994 2024-09-16T05:35:25.004 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-16T05:35:26.005 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T05:35:26.005 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T05:35:26.377 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672994 -lt 42949672994 2024-09-16T05:35:26.377 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:35:26.377 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509469 2024-09-16T05:35:26.377 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:35:26.378 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-09-16T05:35:26.378 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509469 2024-09-16T05:35:26.379 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:35:26.380 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509469 2024-09-16T05:35:26.380 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509469' 2024-09-16T05:35:26.380 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.2 seq 64424509469 2024-09-16T05:35:26.380 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T05:35:26.752 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509469 -lt 64424509469 2024-09-16T05:35:26.753 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:35:26.753 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345938 2024-09-16T05:35:26.753 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:35:26.754 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-09-16T05:35:26.754 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345938 2024-09-16T05:35:26.754 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:35:26.755 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345938 2024-09-16T05:35:26.755 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.3 seq 85899345938 2024-09-16T05:35:26.756 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345938' 2024-09-16T05:35:26.756 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T05:35:27.122 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345938 -lt 85899345938 2024-09-16T05:35:27.122 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: get_num_pgs 2024-09-16T05:35:27.123 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:35:27.123 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:35:27.639 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: test 32 == 0 2024-09-16T05:35:27.639 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_peered: true 2024-09-16T05:35:27.639 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: get_num_active_or_peered 2024-09-16T05:35:27.639 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1394: get_num_active_or_peered: local expression 2024-09-16T05:35:27.639 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-09-16T05:35:27.639 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-09-16T05:35:27.640 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: get_num_active_or_peered: ceph --format json pg dump pgs 2024-09-16T05:35:27.640 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-09-16T05:35:28.002 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: cur_peered=27 2024-09-16T05:35:28.002 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: get_num_pgs 2024-09-16T05:35:28.003 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:35:28.003 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:35:28.531 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: test 27 = 32 2024-09-16T05:35:28.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_peered: test 27 '!=' -1 2024-09-16T05:35:28.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_peered: loop=0 2024-09-16T05:35:28.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_peered: num_peered=27 2024-09-16T05:35:28.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1726: wait_for_peered: eval 2024-09-16T05:35:28.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1727: wait_for_peered: sleep 0.1 2024-09-16T05:35:28.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1728: wait_for_peered: loop+=1 2024-09-16T05:35:28.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_peered: true 2024-09-16T05:35:28.632 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: get_num_active_or_peered 2024-09-16T05:35:28.632 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1394: get_num_active_or_peered: local expression 2024-09-16T05:35:28.633 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-09-16T05:35:28.633 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-09-16T05:35:28.633 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: get_num_active_or_peered: ceph --format json pg dump pgs 2024-09-16T05:35:28.633 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-09-16T05:35:28.996 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: cur_peered=27 2024-09-16T05:35:28.996 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: get_num_pgs 2024-09-16T05:35:28.997 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:35:28.997 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:35:29.560 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: test 27 = 32 2024-09-16T05:35:29.560 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_peered: test 27 '!=' 27 2024-09-16T05:35:29.560 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_peered: get_is_making_recovery_progress 2024-09-16T05:35:29.560 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T05:35:29.560 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T05:35:29.560 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T05:35:29.560 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T05:35:29.560 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T05:35:29.561 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T05:35:30.061 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T05:35:30.061 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T05:35:30.061 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1721: wait_for_peered: (( 1 >= 13 )) 2024-09-16T05:35:30.061 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1726: wait_for_peered: eval 2024-09-16T05:35:30.061 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1727: wait_for_peered: sleep 0.2 2024-09-16T05:35:30.262 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1728: wait_for_peered: loop+=1 2024-09-16T05:35:30.262 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_peered: true 2024-09-16T05:35:30.262 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: get_num_active_or_peered 2024-09-16T05:35:30.263 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1394: get_num_active_or_peered: local expression 2024-09-16T05:35:30.263 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-09-16T05:35:30.263 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-09-16T05:35:30.263 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: get_num_active_or_peered: ceph --format json pg dump pgs 2024-09-16T05:35:30.263 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-09-16T05:35:30.626 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: cur_peered=32 2024-09-16T05:35:30.626 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: get_num_pgs 2024-09-16T05:35:30.627 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:35:30.627 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:35:31.125 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: test 32 = 32 2024-09-16T05:35:31.125 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: break 2024-09-16T05:35:31.125 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1730: wait_for_peered: return 0 2024-09-16T05:35:31.125 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:289: TEST_0_osd: ceph osd ok-to-stop 0 2024-09-16T05:35:31.522 INFO:tasks.workunit.client.0.smithi178.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":0,"num_not_ok_pgs":0} 2024-09-16T05:35:31.533 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:290: TEST_0_osd: ceph osd ok-to-stop 0 1 2024-09-16T05:35:31.889 INFO:tasks.workunit.client.0.smithi178.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T05:35:31.899 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:291: TEST_0_osd: ceph osd ok-to-stop 0 1 2 2024-09-16T05:35:32.152 INFO:tasks.workunit.client.0.smithi178.stdout:{"ok_to_stop":false,"osds":[0,1,2],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T05:35:32.152 INFO:tasks.workunit.client.0.smithi178.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-09-16T05:35:32.155 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:292: TEST_0_osd: ceph osd ok-to-stop 1 2 3 2024-09-16T05:35:32.409 INFO:tasks.workunit.client.0.smithi178.stdout:{"ok_to_stop":false,"osds":[1,2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T05:35:32.410 INFO:tasks.workunit.client.0.smithi178.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-09-16T05:35:32.413 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-16T05:35:32.414 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:35:32.414 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:35:32.414 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:35:32.415 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:35:32.415 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:35:32.523 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:35:32.523 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-16T05:35:32.523 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-16T05:35:32.523 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:35:32.523 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-16T05:35:32.523 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:35:32.523 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:35:32.523 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:35:32.523 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:35:32.524 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:35:32.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:35:32.529 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:35:32.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:35:32.530 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:35:32.531 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:35:32.531 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:35:32.531 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:35:32.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:35:32.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:35:32.533 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:35:32.533 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:35:32.533 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:35:32.534 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:35:32.534 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-16T05:35:32.558 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:35:32.558 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:35:32.558 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:35:32.558 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53882 2024-09-16T05:35:32.560 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:35:32.560 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:35:32.560 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-16T05:35:32.560 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-16T05:35:32.560 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-09-16T05:35:32.560 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-09-16T05:35:32.560 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-16T05:35:32.560 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:35:32.560 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-16T05:35:32.561 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:35:32.561 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:35:32.561 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:35:32.561 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:35:32.561 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:35:32.561 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:35:32.561 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:35:32.562 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:35:32.562 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:35:32.563 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:35:32.564 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:35:32.564 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:35:32.564 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:35:32.564 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:35:32.565 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:35:32.565 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:35:32.565 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:35:32.567 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:35:32.567 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-16T05:35:32.568 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:35:32.568 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:35:32.568 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:35:32.568 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53882 2024-09-16T05:35:32.569 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:35:32.569 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:35:32.569 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-09-16T05:35:32.570 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:35:32.570 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:35:32.570 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:35:32.570 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.53882 2024-09-16T05:35:32.572 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:35:32.572 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:35:32.573 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:35:32.573 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-16T05:35:32.573 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_1_mon_checks td/ok-to-stop 2024-09-16T05:35:32.573 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:28: TEST_1_mon_checks: local dir=td/ok-to-stop 2024-09-16T05:35:32.573 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:30: TEST_1_mon_checks: CEPH_ARGS='--fsid=018ac118-e789-4c01-be89-78aa14fbe62f --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-16T05:35:32.573 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:32: TEST_1_mon_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-16T05:35:32.573 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-16T05:35:32.573 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:35:32.573 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:35:32.573 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:35:32.573 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-09-16T05:35:32.573 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-16T05:35:32.948 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:35:32.948 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:35:32.948 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:35:32.948 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:35:32.948 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:35:32.948 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:35:32.948 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53882/$cluster-$name.asok' 2024-09-16T05:35:32.948 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53882/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-16T05:35:33.123 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:35:33.124 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:35:33.125 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:35:33.125 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:35:33.125 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:35:33.125 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:35:33.125 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:35:33.125 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:35:33.125 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:35:33.126 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:35:33.126 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:35:33.126 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:35:33.126 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53882/ceph-mon.a.asok 2024-09-16T05:35:33.126 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:35:33.126 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53882/ceph-mon.a.asok config get fsid 2024-09-16T05:35:33.205 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:35:33.205 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:35:33.205 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:35:33.205 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:35:33.206 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:35:33.206 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:35:33.206 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:35:33.206 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:35:33.206 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:35:33.206 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:35:33.206 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:35:33.206 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53882/ceph-mon.a.asok 2024-09-16T05:35:33.206 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:35:33.206 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53882/ceph-mon.a.asok config get mon_host 2024-09-16T05:35:33.283 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:34: TEST_1_mon_checks: ceph mon ok-to-stop dne 2024-09-16T05:35:34.063 INFO:tasks.workunit.client.0.smithi178.stderr:invalid mon(s) specified: [dne] 2024-09-16T05:35:34.072 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:35: TEST_1_mon_checks: ceph mon ok-to-stop a 2024-09-16T05:35:34.231 INFO:tasks.workunit.client.0.smithi178.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a] 2024-09-16T05:35:34.233 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:37: TEST_1_mon_checks: ceph mon ok-to-add-offline 2024-09-16T05:35:34.383 INFO:tasks.workunit.client.0.smithi178.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2024-09-16T05:35:34.385 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:39: TEST_1_mon_checks: ceph mon ok-to-rm a 2024-09-16T05:35:34.533 INFO:tasks.workunit.client.0.smithi178.stderr:Error EBUSY: removing mon.a would break quorum 2024-09-16T05:35:34.535 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:40: TEST_1_mon_checks: ceph mon ok-to-rm dne 2024-09-16T05:35:34.723 INFO:tasks.workunit.client.0.smithi178.stderr:mon.dne does not exist 2024-09-16T05:35:34.732 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-16T05:35:34.733 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:35:34.733 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:35:34.733 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:35:34.733 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:35:34.734 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:35:34.839 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:35:34.839 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-16T05:35:34.839 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-16T05:35:34.839 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:35:34.839 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-16T05:35:34.839 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:35:34.839 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:35:34.839 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:35:34.840 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:35:34.840 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:35:34.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:35:34.843 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:35:34.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:35:34.844 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:35:34.845 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:35:34.845 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:35:34.845 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:35:34.846 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:35:34.846 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:35:34.846 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:35:34.846 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:35:34.847 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:35:34.848 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:35:34.848 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-16T05:35:34.849 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:35:34.849 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:35:34.850 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:35:34.850 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53882 2024-09-16T05:35:34.851 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:35:34.851 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:35:34.852 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-16T05:35:34.852 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-16T05:35:34.852 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-09-16T05:35:34.852 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-09-16T05:35:34.852 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-16T05:35:34.852 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:35:34.852 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-16T05:35:34.852 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:35:34.852 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:35:34.852 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:35:34.852 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:35:34.852 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:35:34.853 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:35:34.853 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:35:34.853 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:35:34.854 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:35:34.854 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:35:34.855 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:35:34.855 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:35:34.856 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:35:34.856 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:35:34.856 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:35:34.856 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:35:34.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:35:34.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:35:34.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-16T05:35:34.859 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:35:34.859 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:35:34.859 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:35:34.859 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53882 2024-09-16T05:35:34.860 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:35:34.860 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:35:34.860 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-09-16T05:35:34.861 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:35:34.861 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:35:34.861 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:35:34.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.53882 2024-09-16T05:35:34.863 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:35:34.863 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:35:34.864 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:35:34.864 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-16T05:35:34.864 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_2_mons_checks td/ok-to-stop 2024-09-16T05:35:34.864 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:44: TEST_2_mons_checks: local dir=td/ok-to-stop 2024-09-16T05:35:34.864 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:46: TEST_2_mons_checks: CEPH_ARGS='--fsid=018ac118-e789-4c01-be89-78aa14fbe62f --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151 ' 2024-09-16T05:35:34.864 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:48: TEST_2_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-16T05:35:34.864 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-16T05:35:34.864 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:35:34.864 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:35:34.864 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:35:34.864 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-09-16T05:35:34.864 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-16T05:35:35.240 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:35:35.240 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:35:35.240 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:35:35.240 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:35:35.241 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:35:35.241 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:35:35.241 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53882/$cluster-$name.asok' 2024-09-16T05:35:35.241 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53882/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-16T05:35:35.449 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:35:35.450 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:35:35.450 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:35:35.450 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:35:35.450 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:35:35.450 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:35:35.450 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:35:35.450 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:35:35.450 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:35:35.451 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:35:35.451 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:35:35.451 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:35:35.452 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53882/ceph-mon.a.asok 2024-09-16T05:35:35.452 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:35:35.452 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53882/ceph-mon.a.asok config get fsid 2024-09-16T05:35:35.530 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:35:35.530 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:35:35.530 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:35:35.530 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:35:35.530 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:35:35.530 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:35:35.530 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:35:35.531 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:35:35.531 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:35:35.531 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:35:35.531 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:35:35.531 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53882/ceph-mon.a.asok 2024-09-16T05:35:35.531 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:35:35.531 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53882/ceph-mon.a.asok config get mon_host 2024-09-16T05:35:35.608 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:49: TEST_2_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-09-16T05:35:35.608 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-16T05:35:35.608 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:35:35.608 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-09-16T05:35:35.608 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:35:35.608 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-09-16T05:35:35.608 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-09-16T05:35:35.993 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:35:35.994 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:35:35.994 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:35:35.994 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:35:35.994 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:35:35.994 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:35:35.994 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53882/$cluster-$name.asok' 2024-09-16T05:35:35.994 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53882/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-09-16T05:35:36.186 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:35:36.187 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-09-16T05:35:36.187 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:35:36.188 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-16T05:35:36.188 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:35:36.188 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:35:36.188 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-16T05:35:36.188 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-16T05:35:36.188 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-16T05:35:36.188 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:35:36.188 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:35:36.188 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:35:36.188 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53882/ceph-mon.b.asok 2024-09-16T05:35:36.188 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:35:36.188 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53882/ceph-mon.b.asok config get fsid 2024-09-16T05:35:36.265 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-09-16T05:35:36.266 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:35:36.266 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-16T05:35:36.266 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:35:36.266 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:35:36.266 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-16T05:35:36.266 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-16T05:35:36.266 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-16T05:35:36.266 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:35:36.266 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:35:36.266 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:35:36.267 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53882/ceph-mon.b.asok 2024-09-16T05:35:36.267 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:35:36.267 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53882/ceph-mon.b.asok config get mon_host 2024-09-16T05:35:36.343 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:51: TEST_2_mons_checks: ceph mon ok-to-stop dne 2024-09-16T05:35:39.525 INFO:tasks.workunit.client.0.smithi178.stderr:invalid mon(s) specified: [dne] 2024-09-16T05:35:39.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:52: TEST_2_mons_checks: ceph mon ok-to-stop a 2024-09-16T05:35:39.708 INFO:tasks.workunit.client.0.smithi178.stderr:Error EBUSY: not enough monitors would be available (b) after stopping mons [a] 2024-09-16T05:35:39.710 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:53: TEST_2_mons_checks: ceph mon ok-to-stop b 2024-09-16T05:35:39.909 INFO:tasks.workunit.client.0.smithi178.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [b] 2024-09-16T05:35:39.912 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:54: TEST_2_mons_checks: ceph mon ok-to-stop a b 2024-09-16T05:35:40.160 INFO:tasks.workunit.client.0.smithi178.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a,b] 2024-09-16T05:35:40.162 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:56: TEST_2_mons_checks: ceph mon ok-to-add-offline 2024-09-16T05:35:40.411 INFO:tasks.workunit.client.0.smithi178.stderr:adding another mon that is not yet online will not break quorum 2024-09-16T05:35:40.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:58: TEST_2_mons_checks: ceph mon ok-to-rm a 2024-09-16T05:35:40.626 INFO:tasks.workunit.client.0.smithi178.stderr:safe to remove mon.a 2024-09-16T05:35:40.635 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:59: TEST_2_mons_checks: ceph mon ok-to-rm b 2024-09-16T05:35:40.931 INFO:tasks.workunit.client.0.smithi178.stderr:safe to remove mon.b 2024-09-16T05:35:40.941 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:60: TEST_2_mons_checks: ceph mon ok-to-rm dne 2024-09-16T05:35:41.214 INFO:tasks.workunit.client.0.smithi178.stderr:mon.dne does not exist 2024-09-16T05:35:41.224 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-16T05:35:41.224 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:35:41.224 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:35:41.224 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:35:41.224 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:35:41.225 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:35:41.330 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:35:41.330 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-16T05:35:41.330 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-16T05:35:41.330 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:35:41.331 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-16T05:35:41.331 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:35:41.331 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:35:41.331 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:35:41.331 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:35:41.331 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:35:41.334 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:35:41.334 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:35:41.335 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:35:41.335 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:35:41.336 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:35:41.336 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:35:41.336 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:35:41.337 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:35:41.337 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:35:41.338 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:35:41.338 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:35:41.338 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:35:41.339 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:35:41.339 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-16T05:35:41.341 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:35:41.341 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:35:41.342 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:35:41.342 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53882 2024-09-16T05:35:41.343 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:35:41.343 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:35:41.344 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-16T05:35:41.344 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-16T05:35:41.344 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-09-16T05:35:41.344 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-09-16T05:35:41.344 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-16T05:35:41.344 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:35:41.344 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-16T05:35:41.344 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:35:41.344 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:35:41.344 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:35:41.344 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:35:41.344 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:35:41.345 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:35:41.345 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:35:41.345 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:35:41.346 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:35:41.347 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:35:41.347 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:35:41.347 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:35:41.348 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:35:41.348 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:35:41.348 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:35:41.348 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:35:41.349 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:35:41.350 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:35:41.350 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-16T05:35:41.351 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:35:41.351 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:35:41.351 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:35:41.351 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53882 2024-09-16T05:35:41.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:35:41.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:35:41.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-09-16T05:35:41.353 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:35:41.353 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:35:41.354 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:35:41.354 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.53882 2024-09-16T05:35:41.356 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:35:41.356 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:35:41.356 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:35:41.356 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-16T05:35:41.356 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_3_mons_checks td/ok-to-stop 2024-09-16T05:35:41.356 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:64: TEST_3_mons_checks: local dir=td/ok-to-stop 2024-09-16T05:35:41.356 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:66: TEST_3_mons_checks: CEPH_ARGS='--fsid=018ac118-e789-4c01-be89-78aa14fbe62f --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152 ' 2024-09-16T05:35:41.356 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:68: TEST_3_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-16T05:35:41.356 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-16T05:35:41.356 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:35:41.357 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:35:41.357 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:35:41.357 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-09-16T05:35:41.357 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-16T05:35:41.739 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:35:41.739 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:35:41.739 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:35:41.739 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:35:41.739 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:35:41.739 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:35:41.739 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53882/$cluster-$name.asok' 2024-09-16T05:35:41.739 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53882/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-16T05:35:41.914 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:35:41.915 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:35:41.915 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:35:41.915 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:35:41.915 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:35:41.915 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:35:41.916 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:35:41.916 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:35:41.916 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:35:41.917 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:35:41.917 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:35:41.917 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:35:41.917 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53882/ceph-mon.a.asok 2024-09-16T05:35:41.917 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:35:41.917 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53882/ceph-mon.a.asok config get fsid 2024-09-16T05:35:41.995 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:35:41.995 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:35:41.995 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:35:41.995 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:35:41.995 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:35:41.995 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:35:41.995 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:35:41.995 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:35:41.996 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:35:41.996 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:35:41.996 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:35:41.996 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53882/ceph-mon.a.asok 2024-09-16T05:35:41.996 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:35:41.996 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53882/ceph-mon.a.asok config get mon_host 2024-09-16T05:35:42.072 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:69: TEST_3_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-09-16T05:35:42.072 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-16T05:35:42.073 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:35:42.073 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-09-16T05:35:42.073 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:35:42.073 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-09-16T05:35:42.073 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-09-16T05:35:42.450 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:35:42.450 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:35:42.450 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:35:42.450 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:35:42.450 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:35:42.450 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:35:42.450 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53882/$cluster-$name.asok' 2024-09-16T05:35:42.450 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53882/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-09-16T05:35:42.642 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:35:42.644 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-09-16T05:35:42.644 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:35:42.644 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-16T05:35:42.644 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:35:42.644 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:35:42.644 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-16T05:35:42.644 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-16T05:35:42.644 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-16T05:35:42.644 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:35:42.644 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:35:42.645 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:35:42.645 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53882/ceph-mon.b.asok 2024-09-16T05:35:42.645 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:35:42.645 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53882/ceph-mon.b.asok config get fsid 2024-09-16T05:35:42.722 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-09-16T05:35:42.722 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:35:42.722 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-16T05:35:42.722 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:35:42.722 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:35:42.722 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-16T05:35:42.722 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-16T05:35:42.723 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-16T05:35:42.723 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:35:42.723 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:35:42.723 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:35:42.723 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53882/ceph-mon.b.asok 2024-09-16T05:35:42.723 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:35:42.723 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53882/ceph-mon.b.asok config get mon_host 2024-09-16T05:35:42.800 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:70: TEST_3_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2024-09-16T05:35:42.800 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-16T05:35:42.800 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:35:42.800 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=c 2024-09-16T05:35:42.800 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:35:42.800 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/c 2024-09-16T05:35:42.800 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2024-09-16T05:35:43.186 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:35:43.186 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:35:43.187 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:35:43.187 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:35:43.187 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:35:43.187 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:35:43.187 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53882/$cluster-$name.asok' 2024-09-16T05:35:43.187 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53882/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2024-09-16T05:35:43.412 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:35:43.413 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c fsid 2024-09-16T05:35:43.413 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:35:43.413 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-09-16T05:35:43.414 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:35:43.414 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:35:43.414 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-09-16T05:35:43.414 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-09-16T05:35:43.414 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-09-16T05:35:43.415 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:35:43.415 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:35:43.415 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:35:43.415 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53882/ceph-mon.c.asok 2024-09-16T05:35:43.415 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:35:43.415 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53882/ceph-mon.c.asok config get fsid 2024-09-16T05:35:43.493 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c mon_host 2024-09-16T05:35:43.493 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:35:43.493 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-09-16T05:35:43.493 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:35:43.493 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:35:43.493 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-09-16T05:35:43.493 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-09-16T05:35:43.493 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-09-16T05:35:43.493 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:35:43.493 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:35:43.493 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:35:43.494 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53882/ceph-mon.c.asok 2024-09-16T05:35:43.494 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:35:43.494 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53882/ceph-mon.c.asok config get mon_host 2024-09-16T05:35:43.570 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:71: TEST_3_mons_checks: wait_for_quorum 60 3 2024-09-16T05:35:43.570 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-09-16T05:35:43.570 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=3 2024-09-16T05:35:43.570 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-09-16T05:35:43.570 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 3 ]] 2024-09-16T05:35:43.570 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-09-16T05:35:43.570 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-09-16T05:35:43.571 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1726465003 2024-09-16T05:35:43.571 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:35:43.571 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726464943 -lt 1726465003 ]] 2024-09-16T05:35:43.571 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T05:35:43.572 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:35:46.778 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:35:46.778 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:35:41.375743Z","created":"2024-09-16T05:35:41.375743Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:35:46.778 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:35:41.375743Z","created":"2024-09-16T05:35:41.375743Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:35:46.779 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T05:35:46.781 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-09-16T05:35:46.781 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-16T05:35:46.781 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-09-16T05:35:46.781 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-09-16T05:35:46.781 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-09-16T05:35:46.782 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:73: TEST_3_mons_checks: ceph mon ok-to-stop dne 2024-09-16T05:35:47.069 INFO:tasks.workunit.client.0.smithi178.stderr:invalid mon(s) specified: [dne] 2024-09-16T05:35:47.078 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:74: TEST_3_mons_checks: ceph mon ok-to-stop a 2024-09-16T05:35:47.325 INFO:tasks.workunit.client.0.smithi178.stderr:quorum should be preserved (b,c) after stopping [a] 2024-09-16T05:35:47.335 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:75: TEST_3_mons_checks: ceph mon ok-to-stop b 2024-09-16T05:35:47.553 INFO:tasks.workunit.client.0.smithi178.stderr:quorum should be preserved (a,c) after stopping [b] 2024-09-16T05:35:47.562 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:76: TEST_3_mons_checks: ceph mon ok-to-stop c 2024-09-16T05:35:47.964 INFO:tasks.workunit.client.0.smithi178.stderr:quorum should be preserved (a,b) after stopping [c] 2024-09-16T05:35:47.973 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:77: TEST_3_mons_checks: ceph mon ok-to-stop a b 2024-09-16T05:35:48.191 INFO:tasks.workunit.client.0.smithi178.stderr:Error EBUSY: not enough monitors would be available (c) after stopping mons [a,b] 2024-09-16T05:35:48.193 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:78: TEST_3_mons_checks: ceph mon ok-to-stop b c 2024-09-16T05:35:51.337 INFO:tasks.workunit.client.0.smithi178.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [b,c] 2024-09-16T05:35:51.339 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:79: TEST_3_mons_checks: ceph mon ok-to-stop a b c 2024-09-16T05:35:51.483 INFO:tasks.workunit.client.0.smithi178.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a,b,c] 2024-09-16T05:35:51.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:81: TEST_3_mons_checks: ceph mon ok-to-add-offline 2024-09-16T05:35:51.786 INFO:tasks.workunit.client.0.smithi178.stderr:adding another mon that is not yet online will not break quorum 2024-09-16T05:35:51.795 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:83: TEST_3_mons_checks: ceph mon ok-to-rm a 2024-09-16T05:35:52.094 INFO:tasks.workunit.client.0.smithi178.stderr:safe to remove mon.a 2024-09-16T05:35:52.103 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:84: TEST_3_mons_checks: ceph mon ok-to-rm b 2024-09-16T05:35:52.328 INFO:tasks.workunit.client.0.smithi178.stderr:safe to remove mon.b 2024-09-16T05:35:52.337 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:85: TEST_3_mons_checks: ceph mon ok-to-rm c 2024-09-16T05:35:52.588 INFO:tasks.workunit.client.0.smithi178.stderr:safe to remove mon.c 2024-09-16T05:35:52.597 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:87: TEST_3_mons_checks: kill_daemons td/ok-to-stop KILL mon.b 2024-09-16T05:35:52.598 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:35:52.598 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:35:52.598 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:35:52.598 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:35:52.598 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:35:52.703 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:35:52.704 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:88: TEST_3_mons_checks: wait_for_quorum 60 2 2024-09-16T05:35:52.704 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-09-16T05:35:52.704 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=2 2024-09-16T05:35:52.704 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-09-16T05:35:52.704 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 2 ]] 2024-09-16T05:35:52.704 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-09-16T05:35:52.704 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-09-16T05:35:52.705 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1726465012 2024-09-16T05:35:52.705 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:35:52.705 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726464952 -lt 1726465012 ]] 2024-09-16T05:35:52.705 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-16T05:35:52.706 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:35:52.891 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:35:52.891 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:35:41.375743Z","created":"2024-09-16T05:35:41.375743Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:35:52.892 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-16T05:35:52.892 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:35:41.375743Z","created":"2024-09-16T05:35:41.375743Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:35:52.894 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:35:52.894 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:35:52.894 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:35:52.895 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726464952 -lt 1726465012 ]] 2024-09-16T05:35:52.895 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-16T05:35:52.895 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:35:53.080 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:35:53.080 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:35:41.375743Z","created":"2024-09-16T05:35:41.375743Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:35:53.080 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:35:41.375743Z","created":"2024-09-16T05:35:41.375743Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:35:53.080 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-16T05:35:53.083 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:35:53.083 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:35:53.083 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:35:53.084 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726464953 -lt 1726465012 ]] 2024-09-16T05:35:53.084 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-16T05:35:53.084 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:35:56.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:35:56.276 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:35:41.375743Z","created":"2024-09-16T05:35:41.375743Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:35:56.276 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:35:41.375743Z","created":"2024-09-16T05:35:41.375743Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:35:56.276 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-16T05:35:56.278 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:35:56.279 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:35:56.279 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:35:56.280 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726464956 -lt 1726465012 ]] 2024-09-16T05:35:56.280 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-16T05:35:56.280 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:35:56.464 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:35:56.465 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:35:41.375743Z","created":"2024-09-16T05:35:41.375743Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:35:56.465 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:35:41.375743Z","created":"2024-09-16T05:35:41.375743Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:35:56.465 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-16T05:35:56.467 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:35:56.468 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:35:56.468 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:35:56.469 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726464956 -lt 1726465012 ]] 2024-09-16T05:35:56.469 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-16T05:35:56.469 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:35:56.653 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:35:56.653 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:35:41.375743Z","created":"2024-09-16T05:35:41.375743Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:35:56.653 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-16T05:35:56.653 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:35:41.375743Z","created":"2024-09-16T05:35:41.375743Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:35:56.656 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:35:56.656 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:35:56.656 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:35:56.657 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726464956 -lt 1726465012 ]] 2024-09-16T05:35:56.657 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-16T05:35:56.657 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:35:56.845 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:35:56.845 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:35:41.375743Z","created":"2024-09-16T05:35:41.375743Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:35:56.846 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-16T05:35:56.846 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:35:41.375743Z","created":"2024-09-16T05:35:41.375743Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:35:56.848 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:35:56.848 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:35:56.849 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:35:56.849 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726464956 -lt 1726465012 ]] 2024-09-16T05:35:56.850 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-16T05:35:56.850 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:36:00.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:36:00.041 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:35:41.375743Z","created":"2024-09-16T05:35:41.375743Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:36:00.041 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:35:41.375743Z","created":"2024-09-16T05:35:41.375743Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:36:00.042 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-16T05:36:00.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:36:00.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:36:00.044 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:36:00.045 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726464960 -lt 1726465012 ]] 2024-09-16T05:36:00.045 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-16T05:36:00.045 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:36:00.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:36:00.231 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:35:41.375743Z","created":"2024-09-16T05:35:41.375743Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:36:00.231 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-16T05:36:00.231 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:35:41.375743Z","created":"2024-09-16T05:35:41.375743Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:36:00.234 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:36:00.234 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:36:00.234 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:36:00.235 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726464960 -lt 1726465012 ]] 2024-09-16T05:36:00.235 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-16T05:36:00.235 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:36:00.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:36:00.420 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:35:41.375743Z","created":"2024-09-16T05:35:41.375743Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:36:00.421 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:35:41.375743Z","created":"2024-09-16T05:35:41.375743Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:36:00.421 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-16T05:36:00.423 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:36:00.423 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:36:00.423 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:36:00.424 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726464960 -lt 1726465012 ]] 2024-09-16T05:36:00.424 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-16T05:36:00.425 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:36:00.609 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:36:00.610 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:35:41.375743Z","created":"2024-09-16T05:35:41.375743Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:36:00.610 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:35:41.375743Z","created":"2024-09-16T05:35:41.375743Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:36:00.611 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-16T05:36:00.613 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:36:00.613 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:36:00.613 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:36:00.613 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726464960 -lt 1726465012 ]] 2024-09-16T05:36:00.614 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-16T05:36:00.614 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:36:00.798 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:36:00.798 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:35:41.375743Z","created":"2024-09-16T05:35:41.375743Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:36:00.799 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:35:41.375743Z","created":"2024-09-16T05:35:41.375743Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:36:00.799 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-16T05:36:00.801 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:36:00.801 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:36:00.802 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:36:00.802 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726464960 -lt 1726465012 ]] 2024-09-16T05:36:00.802 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-16T05:36:00.802 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:36:00.989 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:36:00.989 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:35:41.375743Z","created":"2024-09-16T05:35:41.375743Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:36:00.990 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:35:41.375743Z","created":"2024-09-16T05:35:41.375743Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:36:00.990 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-16T05:36:00.992 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:36:00.992 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:36:00.992 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:36:00.993 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726464960 -lt 1726465012 ]] 2024-09-16T05:36:00.993 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-16T05:36:00.994 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:36:08.589 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:36:08.589 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":8,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:35:41.375743Z","created":"2024-09-16T05:35:41.375743Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:36:08.589 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:35:41.375743Z","created":"2024-09-16T05:35:41.375743Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:36:08.589 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-16T05:36:08.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-09-16T05:36:08.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-16T05:36:08.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-09-16T05:36:08.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-09-16T05:36:08.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-09-16T05:36:08.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:90: TEST_3_mons_checks: ceph mon ok-to-stop a 2024-09-16T05:36:11.736 INFO:tasks.workunit.client.0.smithi178.stderr:Error EBUSY: not enough monitors would be available (c) after stopping mons [a] 2024-09-16T05:36:11.738 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:91: TEST_3_mons_checks: ceph mon ok-to-stop b 2024-09-16T05:36:14.926 INFO:tasks.workunit.client.0.smithi178.stderr:quorum should be preserved (a,c) after stopping [b] 2024-09-16T05:36:14.935 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:92: TEST_3_mons_checks: ceph mon ok-to-stop c 2024-09-16T05:36:15.151 INFO:tasks.workunit.client.0.smithi178.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [c] 2024-09-16T05:36:15.153 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:94: TEST_3_mons_checks: ceph mon ok-to-add-offline 2024-09-16T05:36:15.332 INFO:tasks.workunit.client.0.smithi178.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2024-09-16T05:36:15.334 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:96: TEST_3_mons_checks: ceph mon ok-to-rm a 2024-09-16T05:36:15.513 INFO:tasks.workunit.client.0.smithi178.stderr:Error EBUSY: removing mon.a would break quorum 2024-09-16T05:36:15.515 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:97: TEST_3_mons_checks: ceph mon ok-to-rm b 2024-09-16T05:36:15.781 INFO:tasks.workunit.client.0.smithi178.stderr:safe to remove mon.b 2024-09-16T05:36:15.790 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:98: TEST_3_mons_checks: ceph mon ok-to-rm c 2024-09-16T05:36:18.933 INFO:tasks.workunit.client.0.smithi178.stderr:Error EBUSY: removing mon.c would break quorum 2024-09-16T05:36:18.935 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-16T05:36:18.936 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:36:18.936 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:36:18.936 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:36:18.936 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:36:18.936 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:36:19.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:36:19.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-16T05:36:19.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-16T05:36:19.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:36:19.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-16T05:36:19.043 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:36:19.043 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:36:19.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:36:19.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:36:19.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:36:19.047 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:36:19.048 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:36:19.049 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:36:19.049 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:36:19.050 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:36:19.050 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:36:19.050 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:36:19.051 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:36:19.051 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:36:19.052 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:36:19.052 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:36:19.052 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:36:19.053 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:36:19.053 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-16T05:36:19.058 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:36:19.058 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:36:19.058 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:36:19.058 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53882 2024-09-16T05:36:19.060 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:36:19.060 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:36:19.060 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-16T05:36:19.060 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-16T05:36:19.060 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-09-16T05:36:19.060 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-09-16T05:36:19.060 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-16T05:36:19.060 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:36:19.060 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-16T05:36:19.060 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:36:19.061 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:36:19.061 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:36:19.061 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:36:19.061 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:36:19.061 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:36:19.061 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:36:19.062 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:36:19.063 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:36:19.064 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:36:19.064 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:36:19.064 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:36:19.065 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:36:19.065 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:36:19.065 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:36:19.065 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:36:19.066 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:36:19.067 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:36:19.067 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-16T05:36:19.068 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:36:19.068 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:36:19.068 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:36:19.069 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53882 2024-09-16T05:36:19.069 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:36:19.069 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:36:19.069 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-09-16T05:36:19.071 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:36:19.071 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:36:19.071 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:36:19.071 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.53882 2024-09-16T05:36:19.073 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:36:19.073 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:36:19.074 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:36:19.074 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-16T05:36:19.074 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_4_mons_checks td/ok-to-stop 2024-09-16T05:36:19.074 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:102: TEST_4_mons_checks: local dir=td/ok-to-stop 2024-09-16T05:36:19.074 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:104: TEST_4_mons_checks: CEPH_ARGS='--fsid=018ac118-e789-4c01-be89-78aa14fbe62f --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152,127.0.0.1:7153 ' 2024-09-16T05:36:19.074 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:106: TEST_4_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-16T05:36:19.074 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-16T05:36:19.074 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:36:19.074 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:36:19.074 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:36:19.074 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-09-16T05:36:19.074 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-16T05:36:19.456 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:36:19.456 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:36:19.456 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:36:19.456 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:36:19.457 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:36:19.457 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:36:19.457 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53882/$cluster-$name.asok' 2024-09-16T05:36:19.457 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53882/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-16T05:36:19.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:36:19.625 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:36:19.625 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:36:19.625 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:36:19.625 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:36:19.626 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:36:19.626 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:36:19.626 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:36:19.626 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:36:19.626 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:36:19.626 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:36:19.626 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:36:19.626 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53882/ceph-mon.a.asok 2024-09-16T05:36:19.627 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:36:19.627 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53882/ceph-mon.a.asok config get fsid 2024-09-16T05:36:19.710 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:36:19.710 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:36:19.710 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:36:19.710 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:36:19.711 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:36:19.711 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:36:19.711 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:36:19.711 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:36:19.711 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:36:19.711 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:36:19.711 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:36:19.711 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53882/ceph-mon.a.asok 2024-09-16T05:36:19.712 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:36:19.712 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53882/ceph-mon.a.asok config get mon_host 2024-09-16T05:36:19.788 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:107: TEST_4_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-09-16T05:36:19.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-16T05:36:19.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:36:19.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-09-16T05:36:19.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:36:19.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-09-16T05:36:19.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-09-16T05:36:20.158 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:36:20.159 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:36:20.159 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:36:20.159 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:36:20.159 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:36:20.159 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:36:20.159 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53882/$cluster-$name.asok' 2024-09-16T05:36:20.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53882/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-09-16T05:36:20.351 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:36:20.352 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-09-16T05:36:20.352 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:36:20.352 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-16T05:36:20.352 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:36:20.353 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:36:20.353 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-16T05:36:20.353 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-16T05:36:20.353 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-16T05:36:20.354 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:36:20.354 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:36:20.354 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:36:20.354 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53882/ceph-mon.b.asok 2024-09-16T05:36:20.354 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:36:20.354 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53882/ceph-mon.b.asok config get fsid 2024-09-16T05:36:20.433 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-09-16T05:36:20.433 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:36:20.433 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-16T05:36:20.433 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:36:20.433 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:36:20.434 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-16T05:36:20.434 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-16T05:36:20.434 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-16T05:36:20.434 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:36:20.434 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:36:20.434 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:36:20.434 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53882/ceph-mon.b.asok 2024-09-16T05:36:20.434 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:36:20.434 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53882/ceph-mon.b.asok config get mon_host 2024-09-16T05:36:20.511 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:108: TEST_4_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2024-09-16T05:36:20.511 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-16T05:36:20.511 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:36:20.511 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=c 2024-09-16T05:36:20.511 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:36:20.512 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/c 2024-09-16T05:36:20.512 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2024-09-16T05:36:20.870 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:36:20.870 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:36:20.870 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:36:20.870 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:36:20.870 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:36:20.870 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:36:20.870 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53882/$cluster-$name.asok' 2024-09-16T05:36:20.870 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53882/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2024-09-16T05:36:21.071 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:36:21.071 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c fsid 2024-09-16T05:36:21.072 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:36:21.072 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-09-16T05:36:21.072 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:36:21.072 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:36:21.073 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-09-16T05:36:21.073 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-09-16T05:36:21.073 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-09-16T05:36:21.073 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:36:21.073 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:36:21.073 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:36:21.073 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53882/ceph-mon.c.asok 2024-09-16T05:36:21.074 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:36:21.074 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53882/ceph-mon.c.asok config get fsid 2024-09-16T05:36:21.152 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c mon_host 2024-09-16T05:36:21.152 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:36:21.152 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-09-16T05:36:21.152 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:36:21.153 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:36:21.153 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-09-16T05:36:21.153 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-09-16T05:36:21.153 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-09-16T05:36:21.153 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:36:21.153 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:36:21.153 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:36:21.153 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53882/ceph-mon.c.asok 2024-09-16T05:36:21.153 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:36:21.153 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53882/ceph-mon.c.asok config get mon_host 2024-09-16T05:36:21.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:109: TEST_4_mons_checks: run_mon td/ok-to-stop d --public-addr=127.0.0.1:7153 2024-09-16T05:36:21.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-16T05:36:21.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:36:21.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=d 2024-09-16T05:36:21.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:36:21.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/d 2024-09-16T05:36:21.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id d --mkfs --mon-data=td/ok-to-stop/d --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7153 2024-09-16T05:36:21.658 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:36:21.658 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:36:21.658 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:36:21.658 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:36:21.659 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:36:21.659 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:36:21.659 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53882/$cluster-$name.asok' 2024-09-16T05:36:21.659 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/d '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53882/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7153 2024-09-16T05:36:21.866 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:36:21.868 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d fsid 2024-09-16T05:36:21.868 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:36:21.868 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-09-16T05:36:21.868 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:36:21.868 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:36:21.868 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-09-16T05:36:21.868 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-09-16T05:36:21.868 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-09-16T05:36:21.869 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:36:21.869 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:36:21.869 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:36:21.869 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53882/ceph-mon.d.asok 2024-09-16T05:36:21.870 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:36:21.870 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53882/ceph-mon.d.asok config get fsid 2024-09-16T05:36:21.946 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d mon_host 2024-09-16T05:36:21.946 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:36:21.946 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-09-16T05:36:21.946 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:36:21.947 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:36:21.947 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-09-16T05:36:21.947 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-09-16T05:36:21.947 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-09-16T05:36:21.947 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:36:21.947 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:36:21.947 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:36:21.947 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53882/ceph-mon.d.asok 2024-09-16T05:36:21.948 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:36:21.948 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53882/ceph-mon.d.asok config get mon_host 2024-09-16T05:36:22.025 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:110: TEST_4_mons_checks: wait_for_quorum 60 4 2024-09-16T05:36:22.025 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-09-16T05:36:22.025 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=4 2024-09-16T05:36:22.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-09-16T05:36:22.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 4 ]] 2024-09-16T05:36:22.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-09-16T05:36:22.026 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-09-16T05:36:22.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1726465042 2024-09-16T05:36:22.026 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:36:22.027 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726464982 -lt 1726465042 ]] 2024-09-16T05:36:22.027 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-16T05:36:22.027 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:36:25.219 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:36:25.219 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:19.094212Z","created":"2024-09-16T05:36:19.094212Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:36:25.219 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-09-16T05:36:25.219 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:19.094212Z","created":"2024-09-16T05:36:19.094212Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:36:25.224 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-09-16T05:36:25.225 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-16T05:36:25.225 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-09-16T05:36:25.225 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-09-16T05:36:25.225 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-09-16T05:36:25.225 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:112: TEST_4_mons_checks: ceph mon ok-to-stop dne 2024-09-16T05:36:25.488 INFO:tasks.workunit.client.0.smithi178.stderr:invalid mon(s) specified: [dne] 2024-09-16T05:36:25.497 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:113: TEST_4_mons_checks: ceph mon ok-to-stop a 2024-09-16T05:36:25.773 INFO:tasks.workunit.client.0.smithi178.stderr:quorum should be preserved (b,c,d) after stopping [a] 2024-09-16T05:36:25.783 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:114: TEST_4_mons_checks: ceph mon ok-to-stop b 2024-09-16T05:36:26.000 INFO:tasks.workunit.client.0.smithi178.stderr:quorum should be preserved (a,c,d) after stopping [b] 2024-09-16T05:36:26.010 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:115: TEST_4_mons_checks: ceph mon ok-to-stop c 2024-09-16T05:36:26.277 INFO:tasks.workunit.client.0.smithi178.stderr:quorum should be preserved (a,b,d) after stopping [c] 2024-09-16T05:36:26.286 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:116: TEST_4_mons_checks: ceph mon ok-to-stop d 2024-09-16T05:36:26.569 INFO:tasks.workunit.client.0.smithi178.stderr:quorum should be preserved (a,b,c) after stopping [d] 2024-09-16T05:36:26.578 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:117: TEST_4_mons_checks: ceph mon ok-to-stop a b 2024-09-16T05:36:26.806 INFO:tasks.workunit.client.0.smithi178.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [a,b] 2024-09-16T05:36:26.808 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:118: TEST_4_mons_checks: ceph mon ok-to-stop c d 2024-09-16T05:36:27.082 INFO:tasks.workunit.client.0.smithi178.stderr:Error EBUSY: not enough monitors would be available (a,b) after stopping mons [c,d] 2024-09-16T05:36:27.084 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:120: TEST_4_mons_checks: ceph mon ok-to-add-offline 2024-09-16T05:36:27.384 INFO:tasks.workunit.client.0.smithi178.stderr:adding another mon that is not yet online will not break quorum 2024-09-16T05:36:27.394 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:122: TEST_4_mons_checks: ceph mon ok-to-rm a 2024-09-16T05:36:27.702 INFO:tasks.workunit.client.0.smithi178.stderr:safe to remove mon.a 2024-09-16T05:36:27.711 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:123: TEST_4_mons_checks: ceph mon ok-to-rm b 2024-09-16T05:36:27.986 INFO:tasks.workunit.client.0.smithi178.stderr:safe to remove mon.b 2024-09-16T05:36:27.996 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:124: TEST_4_mons_checks: ceph mon ok-to-rm c 2024-09-16T05:36:28.279 INFO:tasks.workunit.client.0.smithi178.stderr:safe to remove mon.c 2024-09-16T05:36:28.288 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:126: TEST_4_mons_checks: kill_daemons td/ok-to-stop KILL mon.a 2024-09-16T05:36:28.289 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:36:28.289 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:36:28.289 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:36:28.289 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:36:28.289 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:36:28.394 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:36:28.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:127: TEST_4_mons_checks: wait_for_quorum 60 3 2024-09-16T05:36:28.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-09-16T05:36:28.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=3 2024-09-16T05:36:28.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-09-16T05:36:28.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 3 ]] 2024-09-16T05:36:28.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-09-16T05:36:28.395 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-09-16T05:36:28.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1726465048 2024-09-16T05:36:28.396 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:36:28.396 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726464988 -lt 1726465048 ]] 2024-09-16T05:36:28.396 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T05:36:28.396 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:36:28.581 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:36:28.581 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:19.094212Z","created":"2024-09-16T05:36:19.094212Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:36:28.582 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:19.094212Z","created":"2024-09-16T05:36:19.094212Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:36:28.582 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T05:36:28.584 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:36:28.584 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:36:28.585 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:36:28.585 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726464988 -lt 1726465048 ]] 2024-09-16T05:36:28.585 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T05:36:28.585 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:36:31.778 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:36:31.778 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:19.094212Z","created":"2024-09-16T05:36:19.094212Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:36:31.778 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:19.094212Z","created":"2024-09-16T05:36:19.094212Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:36:31.778 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T05:36:31.781 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:36:31.781 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:36:31.781 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:36:31.782 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726464991 -lt 1726465048 ]] 2024-09-16T05:36:31.782 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T05:36:31.782 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:36:31.967 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:36:31.967 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:19.094212Z","created":"2024-09-16T05:36:19.094212Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:36:31.968 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:19.094212Z","created":"2024-09-16T05:36:19.094212Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:36:31.968 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T05:36:31.970 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:36:31.971 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:36:31.971 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:36:31.971 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726464991 -lt 1726465048 ]] 2024-09-16T05:36:31.971 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T05:36:31.972 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:36:35.162 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:36:35.163 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:19.094212Z","created":"2024-09-16T05:36:19.094212Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:36:35.163 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:19.094212Z","created":"2024-09-16T05:36:19.094212Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:36:35.163 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T05:36:35.165 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:36:35.165 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:36:35.166 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:36:35.167 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726464995 -lt 1726465048 ]] 2024-09-16T05:36:35.167 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T05:36:35.168 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:36:35.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:36:35.352 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:19.094212Z","created":"2024-09-16T05:36:19.094212Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:36:35.352 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T05:36:35.353 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:19.094212Z","created":"2024-09-16T05:36:19.094212Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:36:35.355 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:36:35.355 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:36:35.355 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:36:35.356 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726464995 -lt 1726465048 ]] 2024-09-16T05:36:35.356 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T05:36:35.356 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:36:35.539 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:36:35.540 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:19.094212Z","created":"2024-09-16T05:36:19.094212Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:36:35.540 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:19.094212Z","created":"2024-09-16T05:36:19.094212Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:36:35.540 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T05:36:35.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:36:35.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:36:35.543 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:36:35.544 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726464995 -lt 1726465048 ]] 2024-09-16T05:36:35.544 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T05:36:35.544 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:36:35.729 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:36:35.729 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:19.094212Z","created":"2024-09-16T05:36:19.094212Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:36:35.729 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:19.094212Z","created":"2024-09-16T05:36:19.094212Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:36:35.730 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T05:36:35.732 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:36:35.732 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:36:35.732 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:36:35.733 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726464995 -lt 1726465048 ]] 2024-09-16T05:36:35.733 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T05:36:35.733 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:36:35.918 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:36:35.918 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:19.094212Z","created":"2024-09-16T05:36:19.094212Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:36:35.919 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:19.094212Z","created":"2024-09-16T05:36:19.094212Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:36:35.919 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T05:36:35.921 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:36:35.921 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:36:35.921 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:36:35.922 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726464995 -lt 1726465048 ]] 2024-09-16T05:36:35.922 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T05:36:35.923 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:36:48.802 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:36:48.802 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":8,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:19.094212Z","created":"2024-09-16T05:36:19.094212Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:36:48.802 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:19.094212Z","created":"2024-09-16T05:36:19.094212Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:36:48.802 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T05:36:48.805 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-09-16T05:36:48.805 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-16T05:36:48.805 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-09-16T05:36:48.805 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-09-16T05:36:48.805 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-09-16T05:36:48.805 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:129: TEST_4_mons_checks: ceph mon ok-to-stop a 2024-09-16T05:36:49.094 INFO:tasks.workunit.client.0.smithi178.stderr:quorum should be preserved (b,c,d) after stopping [a] 2024-09-16T05:36:49.103 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:130: TEST_4_mons_checks: ceph mon ok-to-stop b 2024-09-16T05:36:49.293 INFO:tasks.workunit.client.0.smithi178.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [b] 2024-09-16T05:36:49.295 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:131: TEST_4_mons_checks: ceph mon ok-to-stop c 2024-09-16T05:36:49.528 INFO:tasks.workunit.client.0.smithi178.stderr:Error EBUSY: not enough monitors would be available (b,d) after stopping mons [c] 2024-09-16T05:36:49.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:132: TEST_4_mons_checks: ceph mon ok-to-stop d 2024-09-16T05:36:49.787 INFO:tasks.workunit.client.0.smithi178.stderr:Error EBUSY: not enough monitors would be available (b,c) after stopping mons [d] 2024-09-16T05:36:49.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:134: TEST_4_mons_checks: ceph mon ok-to-add-offline 2024-09-16T05:36:52.974 INFO:tasks.workunit.client.0.smithi178.stderr:adding another mon that is not yet online will not break quorum 2024-09-16T05:36:52.983 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:136: TEST_4_mons_checks: ceph mon ok-to-rm a 2024-09-16T05:36:53.246 INFO:tasks.workunit.client.0.smithi178.stderr:safe to remove mon.a 2024-09-16T05:36:53.255 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:137: TEST_4_mons_checks: ceph mon ok-to-rm b 2024-09-16T05:36:53.515 INFO:tasks.workunit.client.0.smithi178.stderr:safe to remove mon.b 2024-09-16T05:36:53.524 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:138: TEST_4_mons_checks: ceph mon ok-to-rm c 2024-09-16T05:36:53.765 INFO:tasks.workunit.client.0.smithi178.stderr:safe to remove mon.c 2024-09-16T05:36:53.774 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:139: TEST_4_mons_checks: ceph mon ok-to-rm d 2024-09-16T05:36:53.984 INFO:tasks.workunit.client.0.smithi178.stderr:safe to remove mon.d 2024-09-16T05:36:53.994 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-16T05:36:53.994 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:36:53.995 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:36:53.995 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:36:53.995 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:36:53.995 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:36:54.101 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:36:54.102 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-16T05:36:54.102 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-16T05:36:54.102 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:36:54.102 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-16T05:36:54.102 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:36:54.102 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:36:54.102 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:36:54.102 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:36:54.102 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:36:54.106 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:36:54.106 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:36:54.106 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:36:54.107 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:36:54.108 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:36:54.108 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:36:54.108 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:36:54.109 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:36:54.109 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:36:54.109 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:36:54.109 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:36:54.110 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:36:54.111 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:36:54.111 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-16T05:36:54.116 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:36:54.116 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:36:54.116 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:36:54.116 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53882 2024-09-16T05:36:54.119 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:36:54.119 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:36:54.119 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-16T05:36:54.119 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-16T05:36:54.119 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-09-16T05:36:54.120 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-09-16T05:36:54.120 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-16T05:36:54.120 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:36:54.120 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-16T05:36:54.120 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:36:54.120 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:36:54.121 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:36:54.121 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:36:54.121 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:36:54.121 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:36:54.121 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:36:54.122 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:36:54.123 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:36:54.123 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:36:54.123 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:36:54.123 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:36:54.123 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:36:54.123 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:36:54.123 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:36:54.123 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:36:54.123 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:36:54.124 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:36:54.124 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-16T05:36:54.125 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:36:54.125 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:36:54.125 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:36:54.126 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53882 2024-09-16T05:36:54.126 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:36:54.126 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:36:54.126 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-09-16T05:36:54.127 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:36:54.128 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:36:54.128 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:36:54.128 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.53882 2024-09-16T05:36:54.130 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:36:54.130 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:36:54.131 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:36:54.131 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-16T05:36:54.131 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_5_mons_checks td/ok-to-stop 2024-09-16T05:36:54.131 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:143: TEST_5_mons_checks: local dir=td/ok-to-stop 2024-09-16T05:36:54.131 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:145: TEST_5_mons_checks: CEPH_ARGS='--fsid=018ac118-e789-4c01-be89-78aa14fbe62f --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152,127.0.0.1:7153,127.0.0.1:7154 ' 2024-09-16T05:36:54.131 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:147: TEST_5_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-16T05:36:54.131 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-16T05:36:54.131 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:36:54.131 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:36:54.131 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:36:54.131 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-09-16T05:36:54.131 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-16T05:36:54.525 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:36:54.525 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:36:54.525 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:36:54.525 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:36:54.526 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:36:54.526 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:36:54.526 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53882/$cluster-$name.asok' 2024-09-16T05:36:54.526 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53882/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-16T05:36:54.725 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:36:54.726 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:36:54.726 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:36:54.726 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:36:54.726 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:36:54.726 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:36:54.727 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:36:54.727 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:36:54.727 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:36:54.728 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:36:54.728 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:36:54.728 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:36:54.728 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53882/ceph-mon.a.asok 2024-09-16T05:36:54.728 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:36:54.728 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53882/ceph-mon.a.asok config get fsid 2024-09-16T05:36:54.805 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:36:54.806 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:36:54.806 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:36:54.806 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:36:54.806 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:36:54.806 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:36:54.807 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:36:54.807 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:36:54.807 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:36:54.807 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:36:54.807 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:36:54.807 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53882/ceph-mon.a.asok 2024-09-16T05:36:54.807 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:36:54.807 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53882/ceph-mon.a.asok config get mon_host 2024-09-16T05:36:54.883 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:148: TEST_5_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-09-16T05:36:54.884 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-16T05:36:54.884 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:36:54.884 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-09-16T05:36:54.884 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:36:54.884 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-09-16T05:36:54.884 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-09-16T05:36:55.236 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:36:55.236 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:36:55.236 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:36:55.237 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:36:55.237 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:36:55.237 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:36:55.237 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53882/$cluster-$name.asok' 2024-09-16T05:36:55.237 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53882/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-09-16T05:36:55.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:36:55.421 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-09-16T05:36:55.422 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:36:55.422 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-16T05:36:55.422 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:36:55.422 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:36:55.422 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-16T05:36:55.422 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-16T05:36:55.422 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-16T05:36:55.422 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:36:55.422 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:36:55.422 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:36:55.422 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53882/ceph-mon.b.asok 2024-09-16T05:36:55.423 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:36:55.423 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53882/ceph-mon.b.asok config get fsid 2024-09-16T05:36:55.501 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-09-16T05:36:55.501 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:36:55.501 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-16T05:36:55.501 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:36:55.501 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:36:55.501 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-16T05:36:55.501 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-16T05:36:55.501 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-16T05:36:55.501 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:36:55.501 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:36:55.501 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:36:55.502 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53882/ceph-mon.b.asok 2024-09-16T05:36:55.502 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:36:55.502 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53882/ceph-mon.b.asok config get mon_host 2024-09-16T05:36:55.577 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:149: TEST_5_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2024-09-16T05:36:55.577 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-16T05:36:55.577 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:36:55.577 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=c 2024-09-16T05:36:55.578 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:36:55.578 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/c 2024-09-16T05:36:55.578 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2024-09-16T05:36:55.939 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:36:55.939 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:36:55.939 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:36:55.939 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:36:55.939 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:36:55.939 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:36:55.939 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53882/$cluster-$name.asok' 2024-09-16T05:36:55.939 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53882/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2024-09-16T05:36:56.139 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:36:56.141 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c fsid 2024-09-16T05:36:56.141 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:36:56.141 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-09-16T05:36:56.141 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:36:56.141 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:36:56.141 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-09-16T05:36:56.141 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-09-16T05:36:56.141 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-09-16T05:36:56.141 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:36:56.141 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:36:56.141 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:36:56.142 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53882/ceph-mon.c.asok 2024-09-16T05:36:56.142 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:36:56.142 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53882/ceph-mon.c.asok config get fsid 2024-09-16T05:36:56.219 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c mon_host 2024-09-16T05:36:56.219 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:36:56.219 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-09-16T05:36:56.219 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:36:56.219 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:36:56.219 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-09-16T05:36:56.219 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-09-16T05:36:56.220 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-09-16T05:36:56.220 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:36:56.220 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:36:56.220 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:36:56.220 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53882/ceph-mon.c.asok 2024-09-16T05:36:56.220 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:36:56.220 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53882/ceph-mon.c.asok config get mon_host 2024-09-16T05:36:56.296 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:150: TEST_5_mons_checks: run_mon td/ok-to-stop d --public-addr=127.0.0.1:7153 2024-09-16T05:36:56.296 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-16T05:36:56.296 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:36:56.296 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=d 2024-09-16T05:36:56.297 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:36:56.297 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/d 2024-09-16T05:36:56.297 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id d --mkfs --mon-data=td/ok-to-stop/d --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7153 2024-09-16T05:36:56.684 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:36:56.684 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:36:56.684 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:36:56.684 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:36:56.684 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:36:56.684 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:36:56.684 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53882/$cluster-$name.asok' 2024-09-16T05:36:56.684 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/d '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53882/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7153 2024-09-16T05:36:56.910 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:36:56.913 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d fsid 2024-09-16T05:36:56.913 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:36:56.914 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-09-16T05:36:56.914 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:36:56.914 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:36:56.914 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-09-16T05:36:56.914 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-09-16T05:36:56.914 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-09-16T05:36:56.914 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:36:56.914 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:36:56.914 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:36:56.914 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53882/ceph-mon.d.asok 2024-09-16T05:36:56.914 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:36:56.914 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53882/ceph-mon.d.asok config get fsid 2024-09-16T05:36:56.991 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d mon_host 2024-09-16T05:36:56.991 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:36:56.992 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-09-16T05:36:56.992 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:36:56.992 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:36:56.992 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-09-16T05:36:56.992 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-09-16T05:36:56.992 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-09-16T05:36:56.992 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:36:56.992 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:36:56.992 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:36:56.992 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53882/ceph-mon.d.asok 2024-09-16T05:36:56.993 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:36:56.993 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53882/ceph-mon.d.asok config get mon_host 2024-09-16T05:36:57.069 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:151: TEST_5_mons_checks: run_mon td/ok-to-stop e --public-addr=127.0.0.1:7154 2024-09-16T05:36:57.069 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-16T05:36:57.069 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:36:57.069 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=e 2024-09-16T05:36:57.069 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:36:57.070 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/e 2024-09-16T05:36:57.070 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id e --mkfs --mon-data=td/ok-to-stop/e --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7154 2024-09-16T05:36:57.420 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:36:57.420 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:36:57.421 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:36:57.421 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:36:57.421 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:36:57.421 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:36:57.421 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53882/$cluster-$name.asok' 2024-09-16T05:36:57.421 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id e --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/e '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53882/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7154 2024-09-16T05:36:57.630 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:36:57.631 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon e fsid 2024-09-16T05:36:57.632 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:36:57.632 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=e 2024-09-16T05:36:57.632 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:36:57.632 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:36:57.632 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.e 2024-09-16T05:36:57.632 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.e 2024-09-16T05:36:57.632 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.e ']' 2024-09-16T05:36:57.632 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:36:57.632 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:36:57.632 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:36:57.632 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53882/ceph-mon.e.asok 2024-09-16T05:36:57.633 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:36:57.633 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53882/ceph-mon.e.asok config get fsid 2024-09-16T05:36:57.711 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon e mon_host 2024-09-16T05:36:57.711 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:36:57.711 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=e 2024-09-16T05:36:57.711 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:36:57.711 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:36:57.711 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.e 2024-09-16T05:36:57.711 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.e 2024-09-16T05:36:57.711 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.e ']' 2024-09-16T05:36:57.711 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:36:57.711 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:36:57.711 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:36:57.712 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53882/ceph-mon.e.asok 2024-09-16T05:36:57.712 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:36:57.712 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53882/ceph-mon.e.asok config get mon_host 2024-09-16T05:36:57.791 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:152: TEST_5_mons_checks: wait_for_quorum 60 5 2024-09-16T05:36:57.791 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-09-16T05:36:57.792 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=5 2024-09-16T05:36:57.792 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-09-16T05:36:57.792 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 5 ]] 2024-09-16T05:36:57.792 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-09-16T05:36:57.792 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-09-16T05:36:57.792 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1726465077 2024-09-16T05:36:57.792 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:36:57.793 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726465017 -lt 1726465077 ]] 2024-09-16T05:36:57.793 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 5' 2024-09-16T05:36:57.793 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:37:10.082 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:37:10.083 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:10.083 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:10.083 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 5' 2024-09-16T05:37:10.086 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-09-16T05:37:10.086 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-16T05:37:10.086 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-09-16T05:37:10.086 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-09-16T05:37:10.086 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-09-16T05:37:10.086 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:154: TEST_5_mons_checks: ceph mon ok-to-stop dne 2024-09-16T05:37:10.357 INFO:tasks.workunit.client.0.smithi178.stderr:invalid mon(s) specified: [dne] 2024-09-16T05:37:10.366 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:155: TEST_5_mons_checks: ceph mon ok-to-stop a 2024-09-16T05:37:10.657 INFO:tasks.workunit.client.0.smithi178.stderr:quorum should be preserved (b,c,d,e) after stopping [a] 2024-09-16T05:37:10.667 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:156: TEST_5_mons_checks: ceph mon ok-to-stop b 2024-09-16T05:37:10.928 INFO:tasks.workunit.client.0.smithi178.stderr:quorum should be preserved (a,c,d,e) after stopping [b] 2024-09-16T05:37:10.935 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:157: TEST_5_mons_checks: ceph mon ok-to-stop c 2024-09-16T05:37:11.220 INFO:tasks.workunit.client.0.smithi178.stderr:quorum should be preserved (a,b,d,e) after stopping [c] 2024-09-16T05:37:11.229 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:158: TEST_5_mons_checks: ceph mon ok-to-stop d 2024-09-16T05:37:11.531 INFO:tasks.workunit.client.0.smithi178.stderr:quorum should be preserved (a,b,c,e) after stopping [d] 2024-09-16T05:37:11.633 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:159: TEST_5_mons_checks: ceph mon ok-to-stop e 2024-09-16T05:37:11.866 INFO:tasks.workunit.client.0.smithi178.stderr:quorum should be preserved (a,b,c,d) after stopping [e] 2024-09-16T05:37:11.875 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:160: TEST_5_mons_checks: ceph mon ok-to-stop a b 2024-09-16T05:37:12.176 INFO:tasks.workunit.client.0.smithi178.stderr:quorum should be preserved (c,d,e) after stopping [a,b] 2024-09-16T05:37:12.185 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:161: TEST_5_mons_checks: ceph mon ok-to-stop c d 2024-09-16T05:37:12.504 INFO:tasks.workunit.client.0.smithi178.stderr:quorum should be preserved (a,b,e) after stopping [c,d] 2024-09-16T05:37:12.514 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:162: TEST_5_mons_checks: ceph mon ok-to-stop a b c 2024-09-16T05:37:12.798 INFO:tasks.workunit.client.0.smithi178.stderr:Error EBUSY: not enough monitors would be available (d,e) after stopping mons [a,b,c] 2024-09-16T05:37:12.800 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:164: TEST_5_mons_checks: ceph mon ok-to-add-offline 2024-09-16T05:37:13.150 INFO:tasks.workunit.client.0.smithi178.stderr:adding another mon that is not yet online will not break quorum 2024-09-16T05:37:13.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:166: TEST_5_mons_checks: ceph mon ok-to-rm a 2024-09-16T05:37:13.426 INFO:tasks.workunit.client.0.smithi178.stderr:safe to remove mon.a 2024-09-16T05:37:13.436 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:167: TEST_5_mons_checks: ceph mon ok-to-rm b 2024-09-16T05:37:13.737 INFO:tasks.workunit.client.0.smithi178.stderr:safe to remove mon.b 2024-09-16T05:37:13.746 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:168: TEST_5_mons_checks: ceph mon ok-to-rm c 2024-09-16T05:37:14.022 INFO:tasks.workunit.client.0.smithi178.stderr:safe to remove mon.c 2024-09-16T05:37:14.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:169: TEST_5_mons_checks: ceph mon ok-to-rm d 2024-09-16T05:37:14.331 INFO:tasks.workunit.client.0.smithi178.stderr:safe to remove mon.d 2024-09-16T05:37:14.340 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:170: TEST_5_mons_checks: ceph mon ok-to-rm e 2024-09-16T05:37:14.641 INFO:tasks.workunit.client.0.smithi178.stderr:safe to remove mon.e 2024-09-16T05:37:14.651 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:172: TEST_5_mons_checks: kill_daemons td/ok-to-stop KILL mon.a 2024-09-16T05:37:14.652 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:37:14.652 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:37:14.652 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:37:14.652 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:37:14.652 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:37:14.758 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:37:14.758 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:173: TEST_5_mons_checks: wait_for_quorum 60 4 2024-09-16T05:37:14.758 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-09-16T05:37:14.758 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=4 2024-09-16T05:37:14.758 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-09-16T05:37:14.758 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 4 ]] 2024-09-16T05:37:14.758 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-09-16T05:37:14.758 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-09-16T05:37:14.758 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1726465094 2024-09-16T05:37:14.758 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:37:14.759 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726465034 -lt 1726465094 ]] 2024-09-16T05:37:14.759 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-16T05:37:14.759 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:37:14.943 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:37:14.943 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:14.943 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:14.944 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-09-16T05:37:14.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:37:14.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:37:14.946 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:37:14.947 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726465034 -lt 1726465094 ]] 2024-09-16T05:37:14.947 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-16T05:37:14.947 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:37:15.132 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:37:15.133 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:15.133 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-09-16T05:37:15.133 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:15.135 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:37:15.136 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:37:15.136 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:37:15.137 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726465035 -lt 1726465094 ]] 2024-09-16T05:37:15.137 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-16T05:37:15.137 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:37:15.321 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:37:15.321 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:15.322 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-09-16T05:37:15.322 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:15.324 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:37:15.324 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:37:15.324 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:37:15.325 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726465035 -lt 1726465094 ]] 2024-09-16T05:37:15.325 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-16T05:37:15.325 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:37:18.517 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:37:18.517 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:18.518 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:18.518 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-09-16T05:37:18.520 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:37:18.520 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:37:18.520 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:37:18.521 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726465038 -lt 1726465094 ]] 2024-09-16T05:37:18.521 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-16T05:37:18.521 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:37:21.715 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:37:21.715 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:21.716 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-09-16T05:37:21.716 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:21.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:37:21.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:37:21.718 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:37:21.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726465041 -lt 1726465094 ]] 2024-09-16T05:37:21.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-16T05:37:21.719 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:37:21.905 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:37:21.905 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:21.906 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:21.906 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-09-16T05:37:21.908 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:37:21.908 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:37:21.909 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:37:21.909 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726465041 -lt 1726465094 ]] 2024-09-16T05:37:21.909 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-16T05:37:21.910 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:37:22.095 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:37:22.095 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:22.096 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-09-16T05:37:22.096 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:22.098 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:37:22.098 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:37:22.099 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:37:22.100 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726465042 -lt 1726465094 ]] 2024-09-16T05:37:22.100 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-16T05:37:22.100 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:37:35.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:37:35.698 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:35.698 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-09-16T05:37:35.698 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:35.700 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-09-16T05:37:35.700 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-16T05:37:35.700 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-09-16T05:37:35.700 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-09-16T05:37:35.701 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-09-16T05:37:35.701 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:175: TEST_5_mons_checks: ceph mon ok-to-stop a 2024-09-16T05:37:35.963 INFO:tasks.workunit.client.0.smithi178.stderr:quorum should be preserved (b,c,d,e) after stopping [a] 2024-09-16T05:37:35.972 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:176: TEST_5_mons_checks: ceph mon ok-to-stop b 2024-09-16T05:37:36.249 INFO:tasks.workunit.client.0.smithi178.stderr:quorum should be preserved (c,d,e) after stopping [b] 2024-09-16T05:37:36.258 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:177: TEST_5_mons_checks: ceph mon ok-to-stop c 2024-09-16T05:37:36.541 INFO:tasks.workunit.client.0.smithi178.stderr:quorum should be preserved (b,d,e) after stopping [c] 2024-09-16T05:37:36.551 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:178: TEST_5_mons_checks: ceph mon ok-to-stop d 2024-09-16T05:37:36.835 INFO:tasks.workunit.client.0.smithi178.stderr:quorum should be preserved (b,c,e) after stopping [d] 2024-09-16T05:37:36.845 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:179: TEST_5_mons_checks: ceph mon ok-to-stop e 2024-09-16T05:37:37.122 INFO:tasks.workunit.client.0.smithi178.stderr:quorum should be preserved (b,c,d) after stopping [e] 2024-09-16T05:37:37.131 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:181: TEST_5_mons_checks: ceph mon ok-to-add-offline 2024-09-16T05:37:37.421 INFO:tasks.workunit.client.0.smithi178.stderr:adding another mon that is not yet online will not break quorum 2024-09-16T05:37:37.430 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:183: TEST_5_mons_checks: ceph mon ok-to-rm a 2024-09-16T05:37:37.708 INFO:tasks.workunit.client.0.smithi178.stderr:safe to remove mon.a 2024-09-16T05:37:37.717 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:184: TEST_5_mons_checks: ceph mon ok-to-rm b 2024-09-16T05:37:38.027 INFO:tasks.workunit.client.0.smithi178.stderr:safe to remove mon.b 2024-09-16T05:37:38.037 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:185: TEST_5_mons_checks: ceph mon ok-to-rm c 2024-09-16T05:37:41.220 INFO:tasks.workunit.client.0.smithi178.stderr:safe to remove mon.c 2024-09-16T05:37:41.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:186: TEST_5_mons_checks: ceph mon ok-to-rm d 2024-09-16T05:37:41.479 INFO:tasks.workunit.client.0.smithi178.stderr:safe to remove mon.d 2024-09-16T05:37:41.488 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:187: TEST_5_mons_checks: ceph mon ok-to-rm e 2024-09-16T05:37:41.765 INFO:tasks.workunit.client.0.smithi178.stderr:safe to remove mon.e 2024-09-16T05:37:41.774 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:189: TEST_5_mons_checks: kill_daemons td/ok-to-stop KILL mon.e 2024-09-16T05:37:41.775 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:37:41.775 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:37:41.776 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:37:41.776 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:37:41.776 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:37:41.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:37:41.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:190: TEST_5_mons_checks: wait_for_quorum 60 3 2024-09-16T05:37:41.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-09-16T05:37:41.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=3 2024-09-16T05:37:41.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-09-16T05:37:41.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 3 ]] 2024-09-16T05:37:41.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-09-16T05:37:41.881 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-09-16T05:37:41.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1726465121 2024-09-16T05:37:41.882 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:37:41.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726465061 -lt 1726465121 ]] 2024-09-16T05:37:41.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T05:37:41.883 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:37:42.068 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:37:42.068 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:42.068 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:42.068 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T05:37:42.071 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:37:42.071 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:37:42.071 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:37:42.072 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726465062 -lt 1726465121 ]] 2024-09-16T05:37:42.072 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T05:37:42.072 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:37:42.258 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:37:42.258 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:42.259 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T05:37:42.259 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:42.261 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:37:42.261 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:37:42.261 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:37:42.262 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726465062 -lt 1726465121 ]] 2024-09-16T05:37:42.262 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T05:37:42.263 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:37:42.447 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:37:42.447 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:42.448 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T05:37:42.448 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:42.450 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:37:42.450 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:37:42.450 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:37:42.451 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726465062 -lt 1726465121 ]] 2024-09-16T05:37:42.451 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T05:37:42.451 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:37:42.636 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:37:42.636 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:42.637 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:42.637 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T05:37:42.639 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:37:42.640 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:37:42.640 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:37:42.640 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726465062 -lt 1726465121 ]] 2024-09-16T05:37:42.640 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T05:37:42.641 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:37:42.826 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:37:42.827 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:42.827 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T05:37:42.827 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:42.830 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:37:42.830 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:37:42.830 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:37:42.831 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726465062 -lt 1726465121 ]] 2024-09-16T05:37:42.831 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T05:37:42.831 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:37:43.017 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:37:43.017 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:43.017 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T05:37:43.017 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:43.020 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:37:43.020 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:37:43.020 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:37:43.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726465063 -lt 1726465121 ]] 2024-09-16T05:37:43.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T05:37:43.021 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:37:43.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:37:43.208 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:43.208 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:43.209 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T05:37:43.211 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:37:43.211 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:37:43.211 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:37:43.212 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726465063 -lt 1726465121 ]] 2024-09-16T05:37:43.212 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T05:37:43.212 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:37:43.397 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:37:43.397 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:43.398 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:43.398 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T05:37:43.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:37:43.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:37:43.400 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:37:43.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726465063 -lt 1726465121 ]] 2024-09-16T05:37:43.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T05:37:43.401 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:37:43.586 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:37:43.586 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:43.587 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:43.587 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T05:37:43.589 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:37:43.589 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:37:43.589 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:37:43.590 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726465063 -lt 1726465121 ]] 2024-09-16T05:37:43.590 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T05:37:43.591 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:37:46.783 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:37:46.783 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:46.784 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T05:37:46.784 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:46.786 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:37:46.786 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:37:46.786 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:37:46.787 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726465066 -lt 1726465121 ]] 2024-09-16T05:37:46.787 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T05:37:46.787 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:37:49.980 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:37:49.980 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:49.980 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T05:37:49.980 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:49.983 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:37:49.983 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:37:49.983 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:37:49.984 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726465069 -lt 1726465121 ]] 2024-09-16T05:37:49.984 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T05:37:49.984 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:37:50.171 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:37:50.171 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:50.171 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:50.172 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T05:37:50.174 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T05:37:50.174 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T05:37:50.174 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T05:37:50.175 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726465070 -lt 1726465121 ]] 2024-09-16T05:37:50.175 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T05:37:50.175 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T05:37:57.807 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T05:37:57.807 INFO:tasks.workunit.client.0.smithi178.stderr:{"election_epoch":16,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:57.808 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":16,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"018ac118-e789-4c01-be89-78aa14fbe62f","modified":"2024-09-16T05:36:54.149768Z","created":"2024-09-16T05:36:54.149768Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T05:37:57.808 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T05:37:57.812 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-09-16T05:37:57.812 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-16T05:37:57.812 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-09-16T05:37:57.812 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-09-16T05:37:57.812 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-09-16T05:37:57.812 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:192: TEST_5_mons_checks: ceph mon ok-to-stop a 2024-09-16T05:37:58.114 INFO:tasks.workunit.client.0.smithi178.stderr:quorum should be preserved (b,c,d) after stopping [a] 2024-09-16T05:37:58.123 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:193: TEST_5_mons_checks: ceph mon ok-to-stop b 2024-09-16T05:37:58.323 INFO:tasks.workunit.client.0.smithi178.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [b] 2024-09-16T05:37:58.325 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:194: TEST_5_mons_checks: ceph mon ok-to-stop c 2024-09-16T05:37:58.550 INFO:tasks.workunit.client.0.smithi178.stderr:Error EBUSY: not enough monitors would be available (b,d) after stopping mons [c] 2024-09-16T05:37:58.552 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:195: TEST_5_mons_checks: ceph mon ok-to-stop d 2024-09-16T05:37:58.777 INFO:tasks.workunit.client.0.smithi178.stderr:Error EBUSY: not enough monitors would be available (b,c) after stopping mons [d] 2024-09-16T05:37:58.779 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:196: TEST_5_mons_checks: ceph mon ok-to-stop e 2024-09-16T05:37:59.075 INFO:tasks.workunit.client.0.smithi178.stderr:quorum should be preserved (b,c,d) after stopping [e] 2024-09-16T05:37:59.085 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:198: TEST_5_mons_checks: ceph mon ok-to-add-offline 2024-09-16T05:37:59.313 INFO:tasks.workunit.client.0.smithi178.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2024-09-16T05:37:59.315 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:200: TEST_5_mons_checks: ceph mon ok-to-rm a 2024-09-16T05:37:59.612 INFO:tasks.workunit.client.0.smithi178.stderr:safe to remove mon.a 2024-09-16T05:37:59.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:201: TEST_5_mons_checks: ceph mon ok-to-rm b 2024-09-16T05:37:59.877 INFO:tasks.workunit.client.0.smithi178.stderr:Error EBUSY: removing mon.b would break quorum 2024-09-16T05:37:59.879 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:202: TEST_5_mons_checks: ceph mon ok-to-rm c 2024-09-16T05:38:00.023 INFO:tasks.workunit.client.0.smithi178.stderr:Error EBUSY: removing mon.c would break quorum 2024-09-16T05:38:00.025 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:203: TEST_5_mons_checks: ceph mon ok-to-rm d 2024-09-16T05:38:00.198 INFO:tasks.workunit.client.0.smithi178.stderr:Error EBUSY: removing mon.d would break quorum 2024-09-16T05:38:00.200 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:204: TEST_5_mons_checks: ceph mon ok-to-rm e 2024-09-16T05:38:00.474 INFO:tasks.workunit.client.0.smithi178.stderr:safe to remove mon.e 2024-09-16T05:38:00.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-16T05:38:00.485 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:38:00.485 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:38:00.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:38:00.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:38:00.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:38:00.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:38:00.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-16T05:38:00.593 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-16T05:38:00.593 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:38:00.593 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-16T05:38:00.593 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:38:00.593 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:38:00.593 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:38:00.593 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:38:00.593 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:38:00.597 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:38:00.598 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:38:00.598 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:38:00.599 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:38:00.599 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:38:00.600 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:38:00.600 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:38:00.601 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:38:00.601 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:38:00.602 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:38:00.602 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:38:00.603 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:38:00.603 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:38:00.603 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-16T05:38:00.612 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:38:00.612 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:38:00.612 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:38:00.612 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53882 2024-09-16T05:38:00.614 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:38:00.614 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:38:00.614 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-09-16T05:38:00.614 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/ok-to-stop 0 2024-09-16T05:38:00.614 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-16T05:38:00.614 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-16T05:38:00.614 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-16T05:38:00.614 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:38:00.614 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:38:00.614 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:38:00.614 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:38:00.614 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:38:00.615 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:38:00.615 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:38:00.616 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:38:00.616 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:38:00.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:38:00.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:38:00.618 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:38:00.619 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:38:00.619 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:38:00.619 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:38:00.620 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:38:00.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:38:00.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-16T05:38:00.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-16T05:38:00.622 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:38:00.622 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:38:00.622 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53882 2024-09-16T05:38:00.622 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53882 2024-09-16T05:38:00.623 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:38:00.623 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:38:00.623 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-09-16T05:38:00.624 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T05:38:00.624 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T05:38:00.649 INFO:tasks.workunit:Running workunit misc/rados-striper.sh... 2024-09-16T05:38:00.650 DEBUG:teuthology.orchestra.run.smithi178:workunit test misc/rados-striper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ad35db06e8924912f01e32b35576297c0f1352b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh 2024-09-16T05:38:00.711 INFO:tasks.workunit.client.0.smithi178.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-16T05:38:00.717 INFO:tasks.workunit.client.0.smithi178.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-16T05:38:00.717 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-16T05:38:00.717 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-16T05:38:00.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-09-16T05:38:00.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-09-16T05:38:00.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-09-16T05:38:00.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-09-16T05:38:00.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-09-16T05:38:00.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-09-16T05:38:00.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/rados-striper 2024-09-16T05:38:00.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:20: run: local dir=td/rados-striper 2024-09-16T05:38:00.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:21: run: shift 2024-09-16T05:38:00.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:23: run: export CEPH_MON=127.0.0.1:7116 2024-09-16T05:38:00.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:23: run: CEPH_MON=127.0.0.1:7116 2024-09-16T05:38:00.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:24: run: export CEPH_ARGS 2024-09-16T05:38:00.718 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:25: run: uuidgen 2024-09-16T05:38:00.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:25: run: CEPH_ARGS+='--fsid=4f778ef2-5f6a-4c20-abbf-1e47af765ead --auth-supported=none ' 2024-09-16T05:38:00.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7116 ' 2024-09-16T05:38:00.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:29: run: setup td/rados-striper 2024-09-16T05:38:00.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/rados-striper 2024-09-16T05:38:00.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/rados-striper 2024-09-16T05:38:00.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/rados-striper 2024-09-16T05:38:00.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:38:00.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/rados-striper KILL 2024-09-16T05:38:00.719 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:38:00.719 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:38:00.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:38:00.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:38:00.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:38:00.720 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:38:00.720 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:38:00.721 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:38:00.721 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:38:00.721 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:38:00.721 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:38:00.721 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:38:00.721 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:38:00.722 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:38:00.722 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:38:00.722 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:38:00.723 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:38:00.723 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:38:00.723 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/rados-striper 2024-09-16T05:38:00.725 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:38:00.726 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:38:00.726 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70944 2024-09-16T05:38:00.726 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70944 2024-09-16T05:38:00.726 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:38:00.726 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:38:00.726 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/rados-striper 2024-09-16T05:38:00.727 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:38:00.727 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:38:00.727 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70944 2024-09-16T05:38:00.727 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70944 2024-09-16T05:38:00.729 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:38:00.729 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-16T05:38:00.729 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-16T05:38:00.729 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:38:00.729 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/rados-striper 1' TERM HUP INT 2024-09-16T05:38:00.729 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:32: run: run_mon td/rados-striper a 2024-09-16T05:38:00.729 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/rados-striper 2024-09-16T05:38:00.729 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:38:00.729 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:38:00.729 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:38:00.730 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/rados-striper/a 2024-09-16T05:38:00.730 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/rados-striper/a --run-dir=td/rados-striper 2024-09-16T05:38:01.076 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:38:01.076 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:38:01.076 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:38:01.076 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:38:01.077 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:38:01.077 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70944 2024-09-16T05:38:01.077 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70944/$cluster-$name.asok' 2024-09-16T05:38:01.077 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/rados-striper/a '--log-file=td/rados-striper/$name.log' '--admin-socket=/tmp/ceph-asok.70944/$cluster-$name.asok' --mon-cluster-log-file=td/rados-striper/log --run-dir=td/rados-striper '--pid-file=td/rados-striper/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T05:38:01.260 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:38:01.261 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:38:01.262 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:38:01.262 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:38:01.262 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:38:01.262 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:38:01.262 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:38:01.262 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:38:01.262 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:38:01.262 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:38:01.262 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:38:01.262 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70944 2024-09-16T05:38:01.262 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70944/ceph-mon.a.asok 2024-09-16T05:38:01.262 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:38:01.262 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70944/ceph-mon.a.asok config get fsid 2024-09-16T05:38:01.341 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:38:01.341 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:38:01.341 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:38:01.341 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:38:01.342 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:38:01.342 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:38:01.342 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:38:01.342 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:38:01.342 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:38:01.342 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:38:01.342 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70944 2024-09-16T05:38:01.342 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70944/ceph-mon.a.asok 2024-09-16T05:38:01.342 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:38:01.342 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70944/ceph-mon.a.asok config get mon_host 2024-09-16T05:38:01.422 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:33: run: run_osd td/rados-striper 0 2024-09-16T05:38:01.422 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/rados-striper 2024-09-16T05:38:01.422 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:38:01.422 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T05:38:01.422 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:38:01.423 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/rados-striper/0 2024-09-16T05:38:01.423 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4f778ef2-5f6a-4c20-abbf-1e47af765ead --auth-supported=none --mon-host=127.0.0.1:7116 ' 2024-09-16T05:38:01.423 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:38:01.423 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:38:01.423 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:38:01.423 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/rados-striper/0' 2024-09-16T05:38:01.423 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/rados-striper/0/journal' 2024-09-16T05:38:01.423 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:38:01.423 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:38:01.423 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2024-09-16T05:38:01.423 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:38:01.423 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:38:01.423 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:38:01.424 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:38:01.424 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:38:01.424 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70944 2024-09-16T05:38:01.424 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70944/$cluster-$name.asok' 2024-09-16T05:38:01.424 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70944/$cluster-$name.asok' 2024-09-16T05:38:01.424 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:38:01.424 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:38:01.424 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:38:01.424 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2024-09-16T05:38:01.424 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2024-09-16T05:38:01.424 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:38:01.424 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:38:01.424 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:38:01.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:38:01.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:38:01.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:38:01.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/rados-striper/0 2024-09-16T05:38:01.425 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:38:01.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dad96efc-f81a-4dde-adef-b0efa275bb55 2024-09-16T05:38:01.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 dad96efc-f81a-4dde-adef-b0efa275bb55' 2024-09-16T05:38:01.425 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 dad96efc-f81a-4dde-adef-b0efa275bb55 2024-09-16T05:38:01.426 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:38:01.439 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA5xOdm8TkjGhAAG1Nbk+8bGzCsCTZ0KDH3yg== 2024-09-16T05:38:01.439 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA5xOdm8TkjGhAAG1Nbk+8bGzCsCTZ0KDH3yg=="}' 2024-09-16T05:38:01.439 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dad96efc-f81a-4dde-adef-b0efa275bb55 -i td/rados-striper/0/new.json 2024-09-16T05:38:02.292 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:38:02.301 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/rados-striper/0/new.json 2024-09-16T05:38:02.302 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=4f778ef2-5f6a-4c20-abbf-1e47af765ead --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/0 --osd-journal=td/rados-striper/0/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.70944/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA5xOdm8TkjGhAAG1Nbk+8bGzCsCTZ0KDH3yg== --osd-uuid dad96efc-f81a-4dde-adef-b0efa275bb55 2024-09-16T05:38:02.330 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:38:02.328+0000 7f074582c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:38:02.361 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:38:02.360+0000 7f074582c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:38:02.362 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:38:02.361+0000 7f074582c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:38:02.362 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:38:02.361+0000 7f074582c640 -1 bdev(0x562ee6d2bc00 td/rados-striper/0/block) open stat got: (1) Operation not permitted 2024-09-16T05:38:02.362 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:38:02.361+0000 7f074582c640 -1 bluestore(td/rados-striper/0) _read_fsid unparsable uuid 2024-09-16T05:38:07.523 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/rados-striper/0/keyring 2024-09-16T05:38:07.523 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:38:07.524 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T05:38:07.525 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T05:38:07.525 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/rados-striper/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:38:07.724 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:38:07.724 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T05:38:07.724 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=4f778ef2-5f6a-4c20-abbf-1e47af765ead --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/0 --osd-journal=td/rados-striper/0/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.70944/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:38:07.724 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:38:07.725 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:38:07.725 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:38:07.749 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:38:07.747+0000 7f3c7761a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:38:07.760 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:38:07.758+0000 7f3c7761a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:38:07.761 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:38:07.759+0000 7f3c7761a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:38:07.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T05:38:07.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:38:07.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:38:07.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:38:07.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:38:07.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:07.934 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:38:07.934 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:38:07.934 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:07.934 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:38:08.136 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:38:09.137 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:38:09.137 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:09.137 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:38:09.138 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:38:09.138 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:09.138 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:38:09.328 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:38:09.350 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:38:09.349+0000 7f3c7761a640 -1 Falling back to public interface 2024-09-16T05:38:10.329 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:38:10.329 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:10.330 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:38:10.330 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:38:10.330 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:10.330 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:38:10.519 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:38:11.521 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:38:11.521 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:11.521 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:38:11.522 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:38:11.522 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:11.522 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:38:11.713 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:38:12.119 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:38:12.117+0000 7f3c7761a640 -1 osd.0 0 log_to_monitors true 2024-09-16T05:38:12.714 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:38:12.714 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:12.714 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:38:12.715 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:38:12.715 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:12.715 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:38:13.079 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:38:14.081 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:38:14.081 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:14.081 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:38:14.082 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:38:14.082 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:14.082 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:38:14.271 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:38:15.272 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:38:15.272 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:15.272 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:38:15.273 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:38:15.273 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:15.273 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:38:15.463 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:38:16.464 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:38:16.464 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:16.464 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:38:16.465 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:38:16.465 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:16.465 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:38:16.656 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:38:17.658 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:38:17.658 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:17.658 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:38:17.658 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:38:17.658 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:17.659 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:38:17.848 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:38:18.849 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:38:18.849 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:18.850 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:38:18.850 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:38:18.850 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:18.850 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:38:19.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:38:20.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:38:20.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:20.041 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:38:20.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:38:20.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:20.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:38:20.232 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:38:21.234 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:38:21.234 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:21.234 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:38:21.234 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:38:21.235 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:21.235 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:38:21.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:38:22.427 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:38:22.427 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:22.427 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:38:22.427 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:38:22.428 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:22.428 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:38:22.620 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:38:23.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:38:23.622 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:23.622 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:38:23.622 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:38:23.622 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:23.622 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:38:23.811 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:38:24.813 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:38:24.813 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:24.813 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:38:24.813 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:38:24.814 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:24.814 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:38:25.002 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:38:26.004 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:38:26.004 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:26.004 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:38:26.005 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:38:26.005 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:26.005 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:38:26.195 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:38:27.196 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:38:27.197 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:27.197 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:38:27.197 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:38:27.197 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:27.198 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:38:27.389 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:38:28.390 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:38:28.391 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:28.391 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:38:28.391 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:38:28.392 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:28.392 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:38:28.581 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:38:29.582 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:38:29.582 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:29.583 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T05:38:29.583 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T05:38:29.583 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:29.583 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:38:29.773 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:38:30.775 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:38:30.775 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:30.775 INFO:tasks.workunit.client.0.smithi178.stdout:19 2024-09-16T05:38:30.776 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-16T05:38:30.776 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:30.776 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:38:30.962 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:38:31.964 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:38:31.964 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:31.964 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-16T05:38:31.964 INFO:tasks.workunit.client.0.smithi178.stdout:20 2024-09-16T05:38:31.964 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:31.965 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:38:32.155 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:38:33.156 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:38:33.156 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:33.156 INFO:tasks.workunit.client.0.smithi178.stdout:21 2024-09-16T05:38:33.157 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-09-16T05:38:33.157 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:33.157 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:38:33.413 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3091293847,v1:127.0.0.1:6801/3091293847] [v2:127.0.0.1:6802/3091293847,v1:127.0.0.1:6803/3091293847] exists,up dad96efc-f81a-4dde-adef-b0efa275bb55 2024-09-16T05:38:33.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:38:33.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:38:33.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:38:33.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:34: run: run_osd td/rados-striper 1 2024-09-16T05:38:33.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/rados-striper 2024-09-16T05:38:33.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:38:33.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T05:38:33.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:38:33.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/rados-striper/1 2024-09-16T05:38:33.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4f778ef2-5f6a-4c20-abbf-1e47af765ead --auth-supported=none --mon-host=127.0.0.1:7116 ' 2024-09-16T05:38:33.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:38:33.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:38:33.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:38:33.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/rados-striper/1' 2024-09-16T05:38:33.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/rados-striper/1/journal' 2024-09-16T05:38:33.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:38:33.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:38:33.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2024-09-16T05:38:33.418 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:38:33.418 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:38:33.419 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:38:33.419 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:38:33.419 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:38:33.419 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70944 2024-09-16T05:38:33.419 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70944/$cluster-$name.asok' 2024-09-16T05:38:33.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70944/$cluster-$name.asok' 2024-09-16T05:38:33.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:38:33.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:38:33.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:38:33.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2024-09-16T05:38:33.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2024-09-16T05:38:33.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:38:33.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:38:33.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:38:33.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:38:33.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:38:33.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:38:33.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/rados-striper/1 2024-09-16T05:38:33.420 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:38:33.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8c0b7578-5378-41b8-b0bb-aeca00c27870 2024-09-16T05:38:33.421 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 8c0b7578-5378-41b8-b0bb-aeca00c27870' 2024-09-16T05:38:33.421 INFO:tasks.workunit.client.0.smithi178.stdout:add osd1 8c0b7578-5378-41b8-b0bb-aeca00c27870 2024-09-16T05:38:33.421 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:38:33.433 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBZxOdmraPHGRAAFr/uBwV6sU7nBx3G7yUIDQ== 2024-09-16T05:38:33.433 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBZxOdmraPHGRAAFr/uBwV6sU7nBx3G7yUIDQ=="}' 2024-09-16T05:38:33.433 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8c0b7578-5378-41b8-b0bb-aeca00c27870 -i td/rados-striper/1/new.json 2024-09-16T05:38:33.728 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:38:33.737 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/rados-striper/1/new.json 2024-09-16T05:38:33.738 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=4f778ef2-5f6a-4c20-abbf-1e47af765ead --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/1 --osd-journal=td/rados-striper/1/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.70944/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBZxOdmraPHGRAAFr/uBwV6sU7nBx3G7yUIDQ== --osd-uuid 8c0b7578-5378-41b8-b0bb-aeca00c27870 2024-09-16T05:38:33.764 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:38:33.764+0000 7f73d5a0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:38:33.766 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:38:33.766+0000 7f73d5a0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:38:33.767 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:38:33.767+0000 7f73d5a0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:38:33.768 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:38:33.767+0000 7f73d5a0b640 -1 bdev(0x55fcd0d25c00 td/rados-striper/1/block) open stat got: (1) Operation not permitted 2024-09-16T05:38:33.768 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:38:33.767+0000 7f73d5a0b640 -1 bluestore(td/rados-striper/1) _read_fsid unparsable uuid 2024-09-16T05:38:39.426 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/rados-striper/1/keyring 2024-09-16T05:38:39.426 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:38:39.427 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd1 key to auth repository 2024-09-16T05:38:39.428 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T05:38:39.428 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/rados-striper/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:38:39.625 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.1 2024-09-16T05:38:39.626 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T05:38:39.626 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=4f778ef2-5f6a-4c20-abbf-1e47af765ead --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/1 --osd-journal=td/rados-striper/1/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.70944/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:38:39.626 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:38:39.626 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:38:39.626 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:38:39.650 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:38:39.649+0000 7f5f16d4b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:38:39.662 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:38:39.661+0000 7f5f16d4b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:38:39.663 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:38:39.663+0000 7f5f16d4b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:38:39.826 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T05:38:39.826 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:38:39.827 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T05:38:39.827 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:38:39.827 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:38:39.827 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:38:39.827 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:39.828 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:38:39.828 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:39.828 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:38:40.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:38:40.755 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:38:40.755+0000 7f5f16d4b640 -1 Falling back to public interface 2024-09-16T05:38:41.032 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:38:41.033 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:41.033 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:38:41.033 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:38:41.033 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:41.033 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:38:41.223 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:38:42.225 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:38:42.225 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:42.225 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:38:42.226 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:38:42.226 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:42.226 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:38:42.410 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:38:43.411 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:38:43.412 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:38:43.412 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:43.412 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:38:43.412 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:43.412 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:38:43.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:38:43.671 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:38:43.670+0000 7f5f16d4b640 -1 osd.1 0 log_to_monitors true 2024-09-16T05:38:44.593 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:38:44.593 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:44.594 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:38:44.594 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:38:44.594 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:44.594 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:38:44.801 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:38:45.802 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:38:45.802 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:45.802 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:38:45.803 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:38:45.803 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:45.803 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:38:45.994 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:38:46.996 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:38:46.996 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:46.996 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:38:46.996 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:38:46.997 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:46.997 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:38:47.186 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:38:48.187 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:38:48.188 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:48.188 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:38:48.188 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:38:48.189 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:48.189 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:38:48.380 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:38:49.381 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:38:49.381 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:49.382 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:38:49.382 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:38:49.382 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:49.382 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:38:49.572 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:38:50.573 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:38:50.573 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:50.574 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:38:50.574 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:38:50.574 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:50.574 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:38:50.765 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:38:51.766 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:38:51.766 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:51.767 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:38:51.767 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:38:51.767 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:51.767 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:38:51.957 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:38:52.958 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:38:52.959 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:52.959 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:38:52.959 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:38:52.959 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:52.960 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:38:53.149 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:38:54.150 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:38:54.151 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:54.151 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:38:54.151 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:38:54.151 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:54.152 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:38:54.339 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:38:55.341 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:38:55.341 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:55.341 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:38:55.342 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:38:55.342 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:55.342 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:38:55.531 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:38:56.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:38:56.533 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:56.533 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:38:56.533 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:38:56.533 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:56.533 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:38:56.723 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:38:57.724 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:38:57.724 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:57.724 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:38:57.725 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:38:57.725 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:57.725 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:38:57.916 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:38:58.918 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:38:58.918 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:38:58.918 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:38:58.919 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:38:58.919 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:38:58.919 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:38:59.105 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:39:00.107 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:39:00.107 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:39:00.107 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:39:00.107 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:39:00.108 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:39:00.108 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:39:00.299 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:39:01.301 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:39:01.301 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:39:01.301 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T05:39:01.302 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T05:39:01.302 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:39:01.302 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:39:01.491 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:39:02.492 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:39:02.493 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:39:02.493 INFO:tasks.workunit.client.0.smithi178.stdout:19 2024-09-16T05:39:02.494 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-16T05:39:02.494 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:39:02.494 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:39:02.675 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:39:03.676 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:39:03.676 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:39:03.676 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-16T05:39:03.676 INFO:tasks.workunit.client.0.smithi178.stdout:20 2024-09-16T05:39:03.677 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:39:03.677 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:39:03.869 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:39:04.870 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:39:04.870 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:39:04.871 INFO:tasks.workunit.client.0.smithi178.stdout:21 2024-09-16T05:39:04.871 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-09-16T05:39:04.871 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:39:04.871 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:39:05.051 INFO:tasks.workunit.client.0.smithi178.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/2203189914,v1:127.0.0.1:6809/2203189914] [v2:127.0.0.1:6810/2203189914,v1:127.0.0.1:6811/2203189914] exists,up 8c0b7578-5378-41b8-b0bb-aeca00c27870 2024-09-16T05:39:05.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:39:05.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:39:05.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:39:05.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:35: run: run_osd td/rados-striper 2 2024-09-16T05:39:05.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/rados-striper 2024-09-16T05:39:05.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:39:05.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T05:39:05.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:39:05.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/rados-striper/2 2024-09-16T05:39:05.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4f778ef2-5f6a-4c20-abbf-1e47af765ead --auth-supported=none --mon-host=127.0.0.1:7116 ' 2024-09-16T05:39:05.056 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:39:05.056 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:39:05.056 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:39:05.056 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/rados-striper/2' 2024-09-16T05:39:05.056 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/rados-striper/2/journal' 2024-09-16T05:39:05.056 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:39:05.056 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:39:05.056 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2024-09-16T05:39:05.056 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:39:05.056 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:39:05.056 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:39:05.056 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:39:05.057 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:39:05.057 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70944 2024-09-16T05:39:05.057 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70944/$cluster-$name.asok' 2024-09-16T05:39:05.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70944/$cluster-$name.asok' 2024-09-16T05:39:05.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:39:05.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:39:05.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:39:05.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2024-09-16T05:39:05.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2024-09-16T05:39:05.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:39:05.058 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:39:05.058 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:39:05.058 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:39:05.058 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:39:05.058 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:39:05.058 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/rados-striper/2 2024-09-16T05:39:05.058 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:39:05.058 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f71b1e67-3492-4ee4-b89a-8e51d985d4aa 2024-09-16T05:39:05.058 INFO:tasks.workunit.client.0.smithi178.stdout:add osd2 f71b1e67-3492-4ee4-b89a-8e51d985d4aa 2024-09-16T05:39:05.059 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f71b1e67-3492-4ee4-b89a-8e51d985d4aa' 2024-09-16T05:39:05.059 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:39:05.071 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB5xOdmMkY+BBAArqm6m5qZ8ZX/G7VZTxqbxw== 2024-09-16T05:39:05.072 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB5xOdmMkY+BBAArqm6m5qZ8ZX/G7VZTxqbxw=="}' 2024-09-16T05:39:05.072 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f71b1e67-3492-4ee4-b89a-8e51d985d4aa -i td/rados-striper/2/new.json 2024-09-16T05:39:05.356 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:39:05.365 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/rados-striper/2/new.json 2024-09-16T05:39:05.366 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=4f778ef2-5f6a-4c20-abbf-1e47af765ead --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/2 --osd-journal=td/rados-striper/2/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.70944/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB5xOdmMkY+BBAArqm6m5qZ8ZX/G7VZTxqbxw== --osd-uuid f71b1e67-3492-4ee4-b89a-8e51d985d4aa 2024-09-16T05:39:05.393 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:39:05.392+0000 7fdd81018640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:39:05.395 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:39:05.393+0000 7fdd81018640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:39:05.396 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:39:05.394+0000 7fdd81018640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:39:05.396 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:39:05.395+0000 7fdd81018640 -1 bdev(0x558854c01c00 td/rados-striper/2/block) open stat got: (1) Operation not permitted 2024-09-16T05:39:05.396 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:39:05.395+0000 7fdd81018640 -1 bluestore(td/rados-striper/2) _read_fsid unparsable uuid 2024-09-16T05:39:11.291 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/rados-striper/2/keyring 2024-09-16T05:39:11.291 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:39:11.292 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T05:39:11.292 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd2 key to auth repository 2024-09-16T05:39:11.293 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/rados-striper/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:39:11.490 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.2 2024-09-16T05:39:11.491 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T05:39:11.491 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=4f778ef2-5f6a-4c20-abbf-1e47af765ead --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/2 --osd-journal=td/rados-striper/2/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.70944/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:39:11.491 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:39:11.491 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:39:11.491 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:39:11.515 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:39:11.514+0000 7f5124102640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:39:11.525 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:39:11.524+0000 7f5124102640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:39:11.526 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:39:11.525+0000 7f5124102640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:39:11.697 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T05:39:11.697 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:39:11.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T05:39:11.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:39:11.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:39:11.698 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:39:11.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:39:11.699 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:39:11.699 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:39:11.699 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:39:11.908 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:39:12.909 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:39:12.909 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:39:12.910 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:39:12.910 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:39:12.910 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:39:12.910 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:39:13.096 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:39:13.127 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:39:13.127+0000 7f5124102640 -1 Falling back to public interface 2024-09-16T05:39:14.097 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:39:14.097 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:39:14.097 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:39:14.098 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:39:14.098 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:39:14.098 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:39:14.286 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:39:15.288 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:39:15.288 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:39:15.288 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:39:15.289 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:39:15.289 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:39:15.289 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:39:15.477 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:39:15.859 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:39:15.858+0000 7f5124102640 -1 osd.2 0 log_to_monitors true 2024-09-16T05:39:16.479 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:39:16.479 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:39:16.479 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:39:16.480 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:39:16.480 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:39:16.480 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:39:16.724 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:39:17.726 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:39:17.726 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:39:17.726 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:39:17.727 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:39:17.727 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:39:17.727 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:39:17.916 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:39:18.918 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:39:18.918 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:39:18.918 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:39:18.919 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:39:18.919 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:39:18.919 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:39:19.107 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:39:20.108 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:39:20.109 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:39:20.109 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:39:20.109 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:39:20.109 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:39:20.109 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:39:20.301 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:39:21.303 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:39:21.303 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:39:21.303 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:39:21.303 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:39:21.304 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:39:21.304 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:39:21.494 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:39:22.496 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:39:22.496 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:39:22.496 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:39:22.497 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:39:22.497 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:39:22.497 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:39:22.686 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:39:23.688 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:39:23.688 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:39:23.688 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:39:23.689 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:39:23.689 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:39:23.689 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:39:23.878 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:39:24.879 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:39:24.879 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:39:24.879 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:39:24.880 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:39:24.880 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:39:24.880 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:39:25.070 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:39:26.071 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:39:26.071 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:39:26.072 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:39:26.072 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:39:26.072 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:39:26.072 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:39:26.263 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:39:27.264 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:39:27.264 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:39:27.265 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:39:27.265 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:39:27.265 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:39:27.265 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:39:27.453 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:39:28.454 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:39:28.454 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:39:28.454 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:39:28.455 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:39:28.455 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:39:28.455 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:39:28.647 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:39:29.648 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:39:29.648 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:39:29.649 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:39:29.649 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:39:29.649 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:39:29.649 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:39:29.838 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:39:30.840 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:39:30.840 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:39:30.840 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:39:30.841 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:39:30.841 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:39:30.841 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:39:31.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:39:32.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:39:32.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:39:32.032 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:39:32.032 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:39:32.032 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:39:32.032 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:39:32.220 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:39:33.224 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:39:33.224 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:39:33.225 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T05:39:33.225 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T05:39:33.225 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:39:33.226 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:39:33.412 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:39:34.413 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:39:34.413 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:39:34.414 INFO:tasks.workunit.client.0.smithi178.stdout:19 2024-09-16T05:39:34.414 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-16T05:39:34.414 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:39:34.414 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:39:34.605 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:39:35.606 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:39:35.607 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:39:35.607 INFO:tasks.workunit.client.0.smithi178.stdout:20 2024-09-16T05:39:35.607 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-16T05:39:35.608 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:39:35.608 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:39:35.800 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:39:36.802 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:39:36.802 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:39:36.802 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-09-16T05:39:36.802 INFO:tasks.workunit.client.0.smithi178.stdout:21 2024-09-16T05:39:36.803 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:39:36.803 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:39:37.008 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:39:38.010 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:39:38.010 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:39:38.010 INFO:tasks.workunit.client.0.smithi178.stdout:22 2024-09-16T05:39:38.011 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-09-16T05:39:38.011 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:39:38.011 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:39:38.199 INFO:tasks.workunit.client.0.smithi178.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/155749610,v1:127.0.0.1:6817/155749610] [v2:127.0.0.1:6818/155749610,v1:127.0.0.1:6819/155749610] exists,up f71b1e67-3492-4ee4-b89a-8e51d985d4aa 2024-09-16T05:39:38.200 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:39:38.200 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:39:38.200 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:39:38.200 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:36: run: create_rbd_pool 2024-09-16T05:39:38.200 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T05:39:38.397 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' does not exist 2024-09-16T05:39:38.406 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T05:39:38.406 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T05:39:38.745 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' already exists 2024-09-16T05:39:38.754 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T05:39:39.755 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T05:39:42.046 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:39: run: dd if=/dev/urandom of=td/rados-striper/toyfile bs=1234 count=1 2024-09-16T05:39:42.047 INFO:tasks.workunit.client.0.smithi178.stderr:1+0 records in 2024-09-16T05:39:42.048 INFO:tasks.workunit.client.0.smithi178.stderr:1+0 records out 2024-09-16T05:39:42.048 INFO:tasks.workunit.client.0.smithi178.stderr:1234 bytes (1.2 kB, 1.2 KiB) copied, 5.3568e-05 s, 23.0 MB/s 2024-09-16T05:39:42.048 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:42: run: rados --pool rbd --striper put toyfile td/rados-striper/toyfile 2024-09-16T05:39:42.721 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:45: run: rados --pool rbd --striper stat toyfile 2024-09-16T05:39:42.721 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:45: run: cut -d , -f 2 2024-09-16T05:39:42.753 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:46: run: rados --pool rbd stat toyfile.0000000000000000 2024-09-16T05:39:42.753 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:46: run: cut -d , -f 2 2024-09-16T05:39:42.782 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:47: run: echo ' size 1234' 2024-09-16T05:39:42.782 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:48: run: diff -w td/rados-striper/stripedStat td/rados-striper/refstat 2024-09-16T05:39:42.783 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:49: run: diff -w td/rados-striper/stat td/rados-striper/refstat 2024-09-16T05:39:42.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:50: run: rados --pool rbd stat toyfile 2024-09-16T05:39:42.812 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:51: run: grep -q 'No such file or directory' td/rados-striper/staterror 2024-09-16T05:39:42.813 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:54: run: rados --pool rbd --striper get toyfile td/rados-striper/stripedGroup 2024-09-16T05:39:43.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:55: run: diff -w td/rados-striper/toyfile td/rados-striper/stripedGroup 2024-09-16T05:39:43.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:56: run: rados --pool rbd get toyfile.0000000000000000 td/rados-striper/nonSTripedGroup 2024-09-16T05:39:43.085 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:57: run: diff -w td/rados-striper/toyfile td/rados-striper/nonSTripedGroup 2024-09-16T05:39:43.085 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:60: run: rados --pool rbd --striper truncate toyfile 12 2024-09-16T05:39:43.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:61: run: rados --pool rbd --striper stat toyfile 2024-09-16T05:39:43.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:61: run: cut -d , -f 2 2024-09-16T05:39:43.510 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:62: run: rados --pool rbd stat toyfile.0000000000000000 2024-09-16T05:39:43.511 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:62: run: cut -d , -f 2 2024-09-16T05:39:43.537 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:63: run: echo ' size 12' 2024-09-16T05:39:43.537 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:64: run: diff -w td/rados-striper/stripedStat td/rados-striper/reftrunc 2024-09-16T05:39:43.538 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:65: run: diff -w td/rados-striper/stat td/rados-striper/reftrunc 2024-09-16T05:39:43.538 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:69: run: rados --pool rbd --striper setxattr toyfile somexattr somevalue 2024-09-16T05:39:43.657 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:70: run: rados --pool rbd --striper getxattr toyfile somexattr 2024-09-16T05:39:43.685 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:71: run: rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2024-09-16T05:39:43.713 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:72: run: echo somevalue 2024-09-16T05:39:43.713 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:73: run: diff -w td/rados-striper/xattrvalue td/rados-striper/refvalue 2024-09-16T05:39:43.713 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:74: run: diff -w td/rados-striper/xattrvalue2 td/rados-striper/refvalue 2024-09-16T05:39:43.714 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:75: run: rados --pool rbd --striper listxattr toyfile 2024-09-16T05:39:43.742 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:76: run: echo somexattr 2024-09-16T05:39:43.742 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:77: run: diff -w td/rados-striper/xattrlist td/rados-striper/reflist 2024-09-16T05:39:43.743 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:78: run: rados --pool rbd listxattr toyfile.0000000000000000 2024-09-16T05:39:43.743 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:78: run: grep -v striper 2024-09-16T05:39:43.769 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:79: run: diff -w td/rados-striper/xattrlist2 td/rados-striper/reflist 2024-09-16T05:39:43.770 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:80: run: rados --pool rbd --striper rmxattr toyfile somexattr 2024-09-16T05:39:43.891 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:82: run: local 'attr_not_found_str=No data available' 2024-09-16T05:39:43.891 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:83: run: uname 2024-09-16T05:39:43.893 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:83: run: '[' Linux = FreeBSD ']' 2024-09-16T05:39:43.893 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:85: run: expect_failure td/rados-striper 'No data available' rados --pool rbd --striper getxattr toyfile somexattr 2024-09-16T05:39:43.893 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1962: expect_failure: local dir=td/rados-striper 2024-09-16T05:39:43.893 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1963: expect_failure: shift 2024-09-16T05:39:43.893 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1964: expect_failure: local 'expected=No data available' 2024-09-16T05:39:43.893 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1965: expect_failure: shift 2024-09-16T05:39:43.893 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1966: expect_failure: local success 2024-09-16T05:39:43.893 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: expect_failure: rados --pool rbd --striper getxattr toyfile somexattr 2024-09-16T05:39:43.921 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: success=false 2024-09-16T05:39:43.921 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: false 2024-09-16T05:39:43.921 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: grep --quiet 'No data available' td/rados-striper/out 2024-09-16T05:39:43.922 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: return 0 2024-09-16T05:39:43.923 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:87: run: expect_failure td/rados-striper 'No data available' rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2024-09-16T05:39:43.923 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1962: expect_failure: local dir=td/rados-striper 2024-09-16T05:39:43.923 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1963: expect_failure: shift 2024-09-16T05:39:43.923 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1964: expect_failure: local 'expected=No data available' 2024-09-16T05:39:43.923 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1965: expect_failure: shift 2024-09-16T05:39:43.923 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1966: expect_failure: local success 2024-09-16T05:39:43.923 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: expect_failure: rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2024-09-16T05:39:43.950 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: success=false 2024-09-16T05:39:43.950 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: false 2024-09-16T05:39:43.950 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: grep --quiet 'No data available' td/rados-striper/out 2024-09-16T05:39:43.951 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: return 0 2024-09-16T05:39:43.951 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:91: run: rados --pool rbd --striper rm toyfile 2024-09-16T05:39:44.143 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:92: run: expect_failure td/rados-striper 'No such file or directory' rados --pool rbd --striper stat toyfile 2024-09-16T05:39:44.143 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1962: expect_failure: local dir=td/rados-striper 2024-09-16T05:39:44.143 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1963: expect_failure: shift 2024-09-16T05:39:44.143 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1964: expect_failure: local 'expected=No such file or directory' 2024-09-16T05:39:44.143 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1965: expect_failure: shift 2024-09-16T05:39:44.143 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1966: expect_failure: local success 2024-09-16T05:39:44.143 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: expect_failure: rados --pool rbd --striper stat toyfile 2024-09-16T05:39:44.170 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: success=false 2024-09-16T05:39:44.170 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: false 2024-09-16T05:39:44.170 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: grep --quiet 'No such file or directory' td/rados-striper/out 2024-09-16T05:39:44.171 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: return 0 2024-09-16T05:39:44.171 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:94: run: expect_failure td/rados-striper 'No such file or directory' rados --pool rbd stat toyfile.0000000000000000 2024-09-16T05:39:44.171 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1962: expect_failure: local dir=td/rados-striper 2024-09-16T05:39:44.172 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1963: expect_failure: shift 2024-09-16T05:39:44.172 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1964: expect_failure: local 'expected=No such file or directory' 2024-09-16T05:39:44.172 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1965: expect_failure: shift 2024-09-16T05:39:44.172 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1966: expect_failure: local success 2024-09-16T05:39:44.172 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: expect_failure: rados --pool rbd stat toyfile.0000000000000000 2024-09-16T05:39:44.197 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: success=false 2024-09-16T05:39:44.197 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: false 2024-09-16T05:39:44.197 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: grep --quiet 'No such file or directory' td/rados-striper/out 2024-09-16T05:39:44.199 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: return 0 2024-09-16T05:39:44.199 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:98: run: teardown td/rados-striper 2024-09-16T05:39:44.199 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/rados-striper 2024-09-16T05:39:44.199 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:39:44.199 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/rados-striper KILL 2024-09-16T05:39:44.199 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:39:44.199 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:39:44.199 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:39:44.199 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:39:44.200 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:39:44.305 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:39:44.306 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:39:44.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:39:44.307 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:39:44.308 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:39:44.308 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:39:44.308 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:39:44.309 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:39:44.309 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:39:44.309 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:39:44.309 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:39:44.310 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:39:44.311 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:39:44.311 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/rados-striper 2024-09-16T05:39:44.322 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:39:44.323 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:39:44.323 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70944 2024-09-16T05:39:44.323 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70944 2024-09-16T05:39:44.325 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:39:44.325 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:39:44.325 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-09-16T05:39:44.325 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/rados-striper 0 2024-09-16T05:39:44.325 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/rados-striper 2024-09-16T05:39:44.325 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-16T05:39:44.325 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/rados-striper KILL 2024-09-16T05:39:44.325 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:39:44.325 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:39:44.325 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:39:44.325 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:39:44.325 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:39:44.326 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:39:44.326 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:39:44.326 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:39:44.327 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:39:44.328 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:39:44.328 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:39:44.328 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:39:44.329 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:39:44.329 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:39:44.329 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:39:44.329 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:39:44.330 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:39:44.331 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-16T05:39:44.331 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/rados-striper 2024-09-16T05:39:44.332 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:39:44.332 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:39:44.332 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70944 2024-09-16T05:39:44.332 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70944 2024-09-16T05:39:44.334 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:39:44.334 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:39:44.334 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-09-16T05:39:44.334 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T05:39:44.334 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T05:39:44.399 INFO:tasks.workunit:Running workunit misc/test-ceph-helpers.sh... 2024-09-16T05:39:44.399 DEBUG:teuthology.orchestra.run.smithi178:workunit test misc/test-ceph-helpers.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ad35db06e8924912f01e32b35576297c0f1352b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh 2024-09-16T05:39:44.463 INFO:tasks.workunit.client.0.smithi178.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-16T05:39:44.467 INFO:tasks.workunit.client.0.smithi178.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-16T05:39:44.467 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: run_tests: export .:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-16T05:39:44.467 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2328: export: `.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin': not a valid identifier 2024-09-16T05:39:44.467 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: run_tests: export CEPH_MON=127.0.0.1:7109 2024-09-16T05:39:44.467 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: run_tests: CEPH_MON=127.0.0.1:7109 2024-09-16T05:39:44.467 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2331: run_tests: export CEPH_ARGS 2024-09-16T05:39:44.467 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2332: run_tests: uuidgen 2024-09-16T05:39:44.468 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2332: run_tests: CEPH_ARGS+=' --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none ' 2024-09-16T05:39:44.468 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2333: run_tests: CEPH_ARGS+='--mon-host=127.0.0.1:7109 ' 2024-09-16T05:39:44.468 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: run_tests: export CEPH_CONF=/dev/null 2024-09-16T05:39:44.468 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: run_tests: CEPH_CONF=/dev/null 2024-09-16T05:39:44.468 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2336: run_tests: set 2024-09-16T05:39:44.468 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2336: run_tests: sed -n -e 's/^\(test_[0-9a-z_]*\) .*/\1/p' 2024-09-16T05:39:44.471 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2336: run_tests: local 'funcs=test_activate_osd 2024-09-16T05:39:44.471 INFO:tasks.workunit.client.0.smithi178.stderr:test_activate_osd_after_mark_down 2024-09-16T05:39:44.471 INFO:tasks.workunit.client.0.smithi178.stderr:test_activate_osd_skip_benchmark 2024-09-16T05:39:44.471 INFO:tasks.workunit.client.0.smithi178.stderr:test_destroy_osd 2024-09-16T05:39:44.471 INFO:tasks.workunit.client.0.smithi178.stderr:test_display_logs 2024-09-16T05:39:44.471 INFO:tasks.workunit.client.0.smithi178.stderr:test_erasure_code_plugin_exists 2024-09-16T05:39:44.471 INFO:tasks.workunit.client.0.smithi178.stderr:test_expect_failure 2024-09-16T05:39:44.471 INFO:tasks.workunit.client.0.smithi178.stderr:test_flush_pg_stats 2024-09-16T05:39:44.471 INFO:tasks.workunit.client.0.smithi178.stderr:test_get_config 2024-09-16T05:39:44.472 INFO:tasks.workunit.client.0.smithi178.stderr:test_get_is_making_recovery_progress 2024-09-16T05:39:44.472 INFO:tasks.workunit.client.0.smithi178.stderr:test_get_last_scrub_stamp 2024-09-16T05:39:44.472 INFO:tasks.workunit.client.0.smithi178.stderr:test_get_not_primary 2024-09-16T05:39:44.472 INFO:tasks.workunit.client.0.smithi178.stderr:test_get_num_active_clean 2024-09-16T05:39:44.472 INFO:tasks.workunit.client.0.smithi178.stderr:test_get_num_active_or_peered 2024-09-16T05:39:44.472 INFO:tasks.workunit.client.0.smithi178.stderr:test_get_num_pgs 2024-09-16T05:39:44.472 INFO:tasks.workunit.client.0.smithi178.stderr:test_get_op_scheduler 2024-09-16T05:39:44.472 INFO:tasks.workunit.client.0.smithi178.stderr:test_get_osd_id_used_by_pgs 2024-09-16T05:39:44.472 INFO:tasks.workunit.client.0.smithi178.stderr:test_get_osds 2024-09-16T05:39:44.472 INFO:tasks.workunit.client.0.smithi178.stderr:test_get_pg 2024-09-16T05:39:44.472 INFO:tasks.workunit.client.0.smithi178.stderr:test_get_primary 2024-09-16T05:39:44.472 INFO:tasks.workunit.client.0.smithi178.stderr:test_get_timeout_delays 2024-09-16T05:39:44.472 INFO:tasks.workunit.client.0.smithi178.stderr:test_is_clean 2024-09-16T05:39:44.472 INFO:tasks.workunit.client.0.smithi178.stderr:test_kill_daemon 2024-09-16T05:39:44.473 INFO:tasks.workunit.client.0.smithi178.stderr:test_kill_daemons 2024-09-16T05:39:44.473 INFO:tasks.workunit.client.0.smithi178.stderr:test_objectstore_tool 2024-09-16T05:39:44.473 INFO:tasks.workunit.client.0.smithi178.stderr:test_pg_schedule_scrub 2024-09-16T05:39:44.473 INFO:tasks.workunit.client.0.smithi178.stderr:test_pg_scrub 2024-09-16T05:39:44.473 INFO:tasks.workunit.client.0.smithi178.stderr:test_repair 2024-09-16T05:39:44.473 INFO:tasks.workunit.client.0.smithi178.stderr:test_run_in_background 2024-09-16T05:39:44.473 INFO:tasks.workunit.client.0.smithi178.stderr:test_run_mon 2024-09-16T05:39:44.473 INFO:tasks.workunit.client.0.smithi178.stderr:test_run_osd 2024-09-16T05:39:44.473 INFO:tasks.workunit.client.0.smithi178.stderr:test_set_config 2024-09-16T05:39:44.473 INFO:tasks.workunit.client.0.smithi178.stderr:test_setup 2024-09-16T05:39:44.474 INFO:tasks.workunit.client.0.smithi178.stderr:test_teardown 2024-09-16T05:39:44.474 INFO:tasks.workunit.client.0.smithi178.stderr:test_wait_background 2024-09-16T05:39:44.474 INFO:tasks.workunit.client.0.smithi178.stderr:test_wait_for_clean 2024-09-16T05:39:44.474 INFO:tasks.workunit.client.0.smithi178.stderr:test_wait_for_health_ok 2024-09-16T05:39:44.474 INFO:tasks.workunit.client.0.smithi178.stderr:test_wait_for_osd 2024-09-16T05:39:44.474 INFO:tasks.workunit.client.0.smithi178.stderr:test_wait_for_peered 2024-09-16T05:39:44.474 INFO:tasks.workunit.client.0.smithi178.stderr:test_wait_for_scrub 2024-09-16T05:39:44.474 INFO:tasks.workunit.client.0.smithi178.stderr:test_wait_osd_id_used_by_pgs' 2024-09-16T05:39:44.474 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2337: run_tests: local dir=td/ceph-helpers 2024-09-16T05:39:44.474 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T05:39:44.475 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_activate_osd td/ceph-helpers 2024-09-16T05:39:44.475 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:886: test_activate_osd: local dir=td/ceph-helpers 2024-09-16T05:39:44.475 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:888: test_activate_osd: setup td/ceph-helpers 2024-09-16T05:39:44.475 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T05:39:44.475 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T05:39:44.475 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:39:44.475 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:39:44.475 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:39:44.475 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:39:44.475 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:39:44.475 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:39:44.475 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:39:44.476 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:39:44.480 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:39:44.480 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:39:44.480 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:39:44.481 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:39:44.481 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:39:44.481 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:39:44.481 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:39:44.481 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:39:44.481 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:39:44.481 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:39:44.481 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:39:44.481 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:39:44.481 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:39:44.481 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:39:44.481 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:39:44.481 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:39:44.482 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:39:44.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:39:44.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:39:44.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:39:44.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T05:39:44.484 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:39:44.484 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:39:44.484 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:39:44.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T05:39:44.484 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:39:44.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-16T05:39:44.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-16T05:39:44.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:39:44.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T05:39:44.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:890: test_activate_osd: run_mon td/ceph-helpers a 2024-09-16T05:39:44.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T05:39:44.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:39:44.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:39:44.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:39:44.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T05:39:44.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-16T05:39:44.856 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:39:44.856 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:39:44.856 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:39:44.856 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:39:44.856 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:39:44.856 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:39:44.856 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:39:44.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T05:39:45.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:39:45.058 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:39:45.058 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:39:45.059 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:39:45.059 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:39:45.059 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:39:45.059 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:39:45.059 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:39:45.059 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:39:45.059 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:39:45.059 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:39:45.059 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:39:45.059 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:39:45.059 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:39:45.059 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T05:39:45.138 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:39:45.138 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:39:45.138 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:39:45.138 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:39:45.138 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:39:45.138 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:39:45.138 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:39:45.138 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:39:45.138 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:39:45.138 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:39:45.138 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:39:45.139 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:39:45.139 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:39:45.139 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T05:39:45.216 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:891: test_activate_osd: run_mgr td/ceph-helpers x 2024-09-16T05:39:45.216 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T05:39:45.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T05:39:45.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T05:39:45.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T05:39:45.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T05:39:45.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T05:39:46.062 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T05:39:46.062 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:39:46.062 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:39:46.062 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:39:46.062 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:39:46.062 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:39:46.062 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:39:46.062 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:39:46.063 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:39:46.093 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:893: test_activate_osd: run_osd td/ceph-helpers 0 2024-09-16T05:39:46.093 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T05:39:46.094 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:39:46.094 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T05:39:46.094 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:39:46.094 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T05:39:46.094 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T05:39:46.094 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:39:46.094 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:39:46.094 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:39:46.094 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T05:39:46.094 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T05:39:46.094 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:39:46.094 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:39:46.094 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T05:39:46.095 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:39:46.095 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:39:46.095 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:39:46.095 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:39:46.095 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:39:46.095 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:39:46.095 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:39:46.095 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:39:46.095 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:39:46.095 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:39:46.095 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:39:46.095 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T05:39:46.095 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T05:39:46.095 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:39:46.096 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:39:46.096 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:39:46.096 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:39:46.096 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:39:46.096 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:39:46.096 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T05:39:46.096 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:39:46.096 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=04edee96-e805-4f79-abab-55648984d9eb 2024-09-16T05:39:46.096 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 04edee96-e805-4f79-abab-55648984d9eb' 2024-09-16T05:39:46.096 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 04edee96-e805-4f79-abab-55648984d9eb 2024-09-16T05:39:46.097 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:39:46.112 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCixOdmzEGsBhAA9r2jr0s1p5KOj/yCTLlRKQ== 2024-09-16T05:39:46.113 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCixOdmzEGsBhAA9r2jr0s1p5KOj/yCTLlRKQ=="}' 2024-09-16T05:39:46.113 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 04edee96-e805-4f79-abab-55648984d9eb -i td/ceph-helpers/0/new.json 2024-09-16T05:39:46.381 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:39:46.391 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T05:39:46.392 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCixOdmzEGsBhAA9r2jr0s1p5KOj/yCTLlRKQ== --osd-uuid 04edee96-e805-4f79-abab-55648984d9eb 2024-09-16T05:39:46.422 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:39:46.421+0000 7f498bf51640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:39:46.450 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:39:46.450+0000 7f498bf51640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:39:46.451 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:39:46.451+0000 7f498bf51640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:39:46.452 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:39:46.451+0000 7f498bf51640 -1 bdev(0x5567f61d1c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-16T05:39:46.452 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:39:46.451+0000 7f498bf51640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T05:39:51.952 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T05:39:51.952 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:39:51.953 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T05:39:51.954 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T05:39:51.954 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:39:52.506 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:39:52.507 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T05:39:52.507 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:39:52.508 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:39:52.508 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:39:52.508 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:39:52.533 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:39:52.532+0000 7f7bf2afe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:39:52.549 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:39:52.548+0000 7f7bf2afe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:39:52.550 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:39:52.549+0000 7f7bf2afe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:39:52.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T05:39:52.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:39:52.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:39:52.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:39:52.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:39:52.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:39:52.946 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:39:52.947 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:39:52.947 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:39:52.947 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:39:53.354 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:39:53.911 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:39:53.910+0000 7f7bf2afe640 -1 Falling back to public interface 2024-09-16T05:39:54.355 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:39:54.356 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:39:54.356 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:39:54.356 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:39:54.356 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:39:54.356 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:39:54.764 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:39:55.765 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:39:55.765 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:39:55.765 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:39:55.766 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:39:55.766 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:39:55.766 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:39:56.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:39:56.485 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:39:56.484+0000 7f7bf2afe640 -1 osd.0 0 log_to_monitors true 2024-09-16T05:39:57.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:39:57.160 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:39:57.160 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:39:57.160 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:39:57.160 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:39:57.161 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:39:57.547 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:39:58.548 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:39:58.548 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:39:58.548 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:39:58.549 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:39:58.549 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:39:58.549 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:39:58.926 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:39:59.928 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:39:59.928 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:39:59.928 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:39:59.929 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:39:59.929 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:39:59.929 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:40:00.367 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:40:01.368 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:40:01.368 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:40:01.369 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:40:01.369 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:40:01.369 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:40:01.369 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:40:01.772 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:40:02.774 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:40:02.774 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:40:02.775 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:40:02.775 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:40:02.775 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:40:02.775 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:40:03.274 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:40:04.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:40:04.276 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:40:04.276 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:40:04.277 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:40:04.277 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:40:04.277 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:40:04.685 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:40:05.686 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:40:05.686 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:40:05.686 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:40:05.687 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:40:05.687 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:40:05.687 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:40:06.226 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:40:07.228 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:40:07.228 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:40:07.228 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:40:07.229 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:40:07.229 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:40:07.229 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:40:07.697 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:40:08.699 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:40:08.699 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:40:08.700 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:40:08.700 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:40:08.700 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:40:08.700 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:40:09.195 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:40:10.196 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:40:10.197 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:40:10.197 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:40:10.197 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:40:10.197 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:40:10.198 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:40:11.017 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:40:12.018 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:40:12.018 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:40:12.019 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:40:12.019 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:40:12.019 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:40:12.019 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:40:12.505 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:40:13.507 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:40:13.507 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:40:13.507 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:40:13.508 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:40:13.508 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:40:13.508 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:40:14.256 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:40:15.258 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:40:15.258 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:40:15.258 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:40:15.259 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:40:15.259 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:40:15.259 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:40:15.968 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:40:16.970 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:40:16.970 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:40:16.970 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:40:16.971 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:40:16.971 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:40:16.971 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:40:17.821 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:40:18.822 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:40:18.822 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:40:18.823 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:40:18.823 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:40:18.823 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:40:18.823 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:40:19.210 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:40:20.211 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:40:20.211 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:40:20.211 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T05:40:20.212 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T05:40:20.212 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:40:20.212 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:40:20.622 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1295508844,v1:127.0.0.1:6803/1295508844] [v2:127.0.0.1:6804/1295508844,v1:127.0.0.1:6805/1295508844] exists,up 04edee96-e805-4f79-abab-55648984d9eb 2024-09-16T05:40:20.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:40:20.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:40:20.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:40:20.624 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: get_asok_path osd.0 2024-09-16T05:40:20.624 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:40:20.624 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:40:20.625 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:40:20.625 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:40:20.625 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:40:20.625 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-osd.0.asok 2024-09-16T05:40:20.625 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: CEPH_ARGS= 2024-09-16T05:40:20.625 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: ceph --format=json daemon /tmp/ceph-asok.77187/ceph-osd.0.asok config get osd_max_backfills 2024-09-16T05:40:20.707 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: local 'backfills={"osd_max_backfills":"1"}' 2024-09-16T05:40:20.707 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:896: test_activate_osd: echo '{"osd_max_backfills":"1"}' 2024-09-16T05:40:20.707 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:896: test_activate_osd: grep --quiet osd_max_backfills 2024-09-16T05:40:20.708 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:898: test_activate_osd: kill_daemons td/ceph-helpers TERM osd 2024-09-16T05:40:20.708 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:40:20.709 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:40:20.709 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:40:20.709 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:40:20.709 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:40:21.017 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:40:21.018 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:900: test_activate_osd: activate_osd td/ceph-helpers 0 --osd-max-backfills 20 2024-09-16T05:40:21.018 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-09-16T05:40:21.018 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T05:40:21.018 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T05:40:21.018 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T05:40:21.018 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-09-16T05:40:21.018 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T05:40:21.018 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:40:21.018 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:40:21.018 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:40:21.018 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T05:40:21.019 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T05:40:21.019 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T05:40:21.019 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T05:40:21.019 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T05:40:21.019 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T05:40:21.019 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:40:21.019 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:40:21.019 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:40:21.019 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:40:21.019 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:40:21.019 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:40:21.019 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:40:21.019 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:40:21.020 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T05:40:21.020 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T05:40:21.020 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:40:21.020 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:40:21.020 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:40:21.020 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:40:21.020 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T05:40:21.020 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-max-backfills 20' 2024-09-16T05:40:21.020 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-09-16T05:40:21.020 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T05:40:21.021 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:40:21.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2024-09-16T05:40:21.021 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-09-16T05:40:21.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T05:40:21.022 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T05:40:21.022 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T05:40:21.022 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T05:40:21.044 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:40:21.043+0000 7f0f39818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:40:21.064 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:40:21.064+0000 7f0f39818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:40:21.065 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:40:21.065+0000 7f0f39818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:40:21.391 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T05:40:21.391 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:40:21.391 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:40:21.392 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:40:21.392 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:40:21.392 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:40:21.392 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:40:21.392 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:40:21.393 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:40:21.393 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:40:21.802 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:40:22.162 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:40:22.161+0000 7f0f39818640 -1 Falling back to public interface 2024-09-16T05:40:22.803 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:40:22.804 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:40:22.804 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:40:22.804 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:40:22.804 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:40:22.805 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:40:23.221 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:40:24.222 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:40:24.223 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:40:24.223 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:40:24.223 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:40:24.223 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:40:24.223 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:40:24.636 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:40:25.594 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:40:25.593+0000 7f0f39818640 -1 osd.0 6 log_to_monitors true 2024-09-16T05:40:25.637 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:40:25.638 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:40:25.638 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:40:25.638 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:40:25.638 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:40:25.638 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:40:26.051 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/1792066044,v1:127.0.0.1:6803/1792066044] [v2:127.0.0.1:6804/1792066044,v1:127.0.0.1:6805/1792066044] exists,up 04edee96-e805-4f79-abab-55648984d9eb 2024-09-16T05:40:26.052 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:40:26.053 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:40:26.053 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:40:26.053 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: get_asok_path osd.0 2024-09-16T05:40:26.053 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:40:26.053 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:40:26.053 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:40:26.053 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:40:26.053 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:40:26.053 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-osd.0.asok 2024-09-16T05:40:26.053 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: CEPH_ARGS= 2024-09-16T05:40:26.053 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: ceph --format=json daemon /tmp/ceph-asok.77187/ceph-osd.0.asok config get osd_max_backfills 2024-09-16T05:40:26.137 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: local 'backfills={"osd_max_backfills":"20"}' 2024-09-16T05:40:26.138 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:903: test_activate_osd: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2024-09-16T05:40:26.138 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:905: test_activate_osd: teardown td/ceph-helpers 2024-09-16T05:40:26.138 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:40:26.138 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:40:26.138 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:40:26.138 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:40:26.138 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:40:26.138 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:40:26.138 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:40:26.138 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:40:26.243 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:40:26.243 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:40:26.243 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:40:26.244 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:40:26.245 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:40:26.245 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:40:26.245 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:40:26.246 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:40:26.246 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:40:26.246 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:40:26.246 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:40:26.247 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:40:26.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:40:26.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:40:26.253 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:40:26.254 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:40:26.254 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:40:26.254 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:40:26.255 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:40:26.256 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:40:26.256 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T05:40:26.256 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_activate_osd_after_mark_down td/ceph-helpers 2024-09-16T05:40:26.256 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:909: test_activate_osd_after_mark_down: local dir=td/ceph-helpers 2024-09-16T05:40:26.256 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:911: test_activate_osd_after_mark_down: setup td/ceph-helpers 2024-09-16T05:40:26.256 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T05:40:26.256 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T05:40:26.256 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:40:26.256 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:40:26.256 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:40:26.256 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:40:26.256 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:40:26.256 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:40:26.257 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:40:26.257 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:40:26.257 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:40:26.257 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:40:26.257 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:40:26.258 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:40:26.258 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:40:26.258 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:40:26.259 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:40:26.259 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:40:26.259 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:40:26.260 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:40:26.260 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:40:26.260 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:40:26.262 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:40:26.262 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:40:26.262 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:40:26.263 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:40:26.263 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:40:26.263 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:40:26.263 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:40:26.263 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:40:26.263 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T05:40:26.265 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:40:26.265 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:40:26.265 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:40:26.265 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T05:40:26.266 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:40:26.267 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:40:26.267 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:40:26.267 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T05:40:26.267 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:913: test_activate_osd_after_mark_down: run_mon td/ceph-helpers a 2024-09-16T05:40:26.267 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T05:40:26.267 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:40:26.267 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:40:26.267 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:40:26.267 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T05:40:26.267 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-16T05:40:26.625 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:40:26.626 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:40:26.626 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:40:26.626 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:40:26.626 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:40:26.626 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:40:26.626 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:40:26.626 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T05:40:26.817 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:40:26.818 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:40:26.818 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:40:26.818 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:40:26.818 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:40:26.819 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:40:26.819 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:40:26.819 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:40:26.819 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:40:26.819 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:40:26.819 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:40:26.819 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:40:26.819 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:40:26.820 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:40:26.820 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T05:40:26.898 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:40:26.898 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:40:26.898 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:40:26.898 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:40:26.898 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:40:26.898 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:40:26.898 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:40:26.898 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:40:26.898 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:40:26.898 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:40:26.898 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:40:26.899 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:40:26.899 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:40:26.899 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T05:40:26.975 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: test_activate_osd_after_mark_down: run_mgr td/ceph-helpers x 2024-09-16T05:40:26.975 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T05:40:26.975 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T05:40:26.975 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T05:40:26.976 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T05:40:26.976 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T05:40:26.976 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T05:40:27.822 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T05:40:27.822 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:40:27.822 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:40:27.822 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:40:27.822 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:40:27.822 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:40:27.822 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:40:27.822 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:40:27.823 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:40:27.853 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: test_activate_osd_after_mark_down: run_osd td/ceph-helpers 0 2024-09-16T05:40:27.854 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T05:40:27.854 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:40:27.854 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T05:40:27.854 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:40:27.854 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T05:40:27.854 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T05:40:27.854 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:40:27.854 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:40:27.854 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:40:27.854 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T05:40:27.855 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T05:40:27.855 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:40:27.855 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:40:27.855 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T05:40:27.855 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:40:27.855 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:40:27.855 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:40:27.855 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:40:27.855 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:40:27.855 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:40:27.855 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:40:27.855 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:40:27.855 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:40:27.855 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:40:27.856 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:40:27.856 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T05:40:27.856 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T05:40:27.856 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:40:27.856 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:40:27.856 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:40:27.856 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:40:27.856 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:40:27.856 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:40:27.856 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T05:40:27.856 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:40:27.856 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a93d35b5-7d15-49ef-867b-a76ed87911e5 2024-09-16T05:40:27.856 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a93d35b5-7d15-49ef-867b-a76ed87911e5' 2024-09-16T05:40:27.857 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 a93d35b5-7d15-49ef-867b-a76ed87911e5 2024-09-16T05:40:27.857 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:40:27.873 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDLxOdm5qT8MxAAKo1CJFxkC+dynn6VKjPQoA== 2024-09-16T05:40:27.873 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDLxOdm5qT8MxAAKo1CJFxkC+dynn6VKjPQoA=="}' 2024-09-16T05:40:27.873 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a93d35b5-7d15-49ef-867b-a76ed87911e5 -i td/ceph-helpers/0/new.json 2024-09-16T05:40:28.142 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:40:28.151 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T05:40:28.152 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDLxOdm5qT8MxAAKo1CJFxkC+dynn6VKjPQoA== --osd-uuid a93d35b5-7d15-49ef-867b-a76ed87911e5 2024-09-16T05:40:28.181 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:40:28.180+0000 7f253cceb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:40:28.211 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:40:28.210+0000 7f253cceb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:40:28.212 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:40:28.211+0000 7f253cceb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:40:28.212 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:40:28.212+0000 7f253cceb640 -1 bdev(0x560cd5127c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-16T05:40:28.213 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:40:28.212+0000 7f253cceb640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T05:40:33.702 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T05:40:33.702 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:40:33.703 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T05:40:33.703 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T05:40:33.703 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:40:34.225 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:40:34.226 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T05:40:34.226 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:40:34.226 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:40:34.226 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:40:34.226 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:40:34.252 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:40:34.251+0000 7f0383816640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:40:34.273 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:40:34.273+0000 7f0383816640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:40:34.274 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:40:34.274+0000 7f0383816640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:40:34.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T05:40:34.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:40:34.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:40:34.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:40:34.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:40:34.662 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:40:34.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:40:34.663 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:40:34.663 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:40:34.663 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:40:35.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:40:35.868 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:40:35.867+0000 7f0383816640 -1 Falling back to public interface 2024-09-16T05:40:36.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:40:36.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:40:36.043 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:40:36.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:40:36.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:40:36.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:40:36.464 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:40:37.465 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:40:37.465 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:40:37.465 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:40:37.466 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:40:37.466 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:40:37.466 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:40:37.856 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:40:38.527 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:40:38.526+0000 7f0383816640 -1 osd.0 0 log_to_monitors true 2024-09-16T05:40:38.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:40:38.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:40:38.858 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:40:38.859 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:40:38.859 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:40:38.859 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:40:39.239 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:40:40.241 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:40:40.241 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:40:40.241 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:40:40.241 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:40:40.242 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:40:40.242 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:40:40.642 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:40:41.643 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:40:41.643 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:40:41.644 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:40:41.644 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:40:41.644 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:40:41.644 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:40:42.067 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:40:43.069 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:40:43.069 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:40:43.069 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:40:43.070 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:40:43.070 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:40:43.070 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:40:43.524 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:40:44.525 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:40:44.525 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:40:44.526 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:40:44.526 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:40:44.526 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:40:44.526 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:40:44.911 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:40:45.913 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:40:45.913 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:40:45.913 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:40:45.914 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:40:45.914 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:40:45.914 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:40:46.370 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:40:47.371 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:40:47.371 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:40:47.372 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:40:47.372 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:40:47.372 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:40:47.373 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:40:47.776 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:40:48.777 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:40:48.780 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:40:48.780 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:40:48.780 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:40:48.781 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:40:48.781 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:40:49.273 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:40:50.274 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:40:50.274 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:40:50.274 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:40:50.274 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:40:50.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:40:50.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:40:51.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:40:52.160 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:40:52.160 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:40:52.161 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:40:52.161 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:40:52.161 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:40:52.161 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:40:52.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:40:54.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:40:54.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:40:54.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:40:54.001 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:40:54.001 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:40:54.001 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:40:54.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:40:55.790 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:40:55.790 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:40:55.790 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:40:55.791 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:40:55.791 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:40:55.791 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:40:56.674 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:40:57.676 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:40:57.676 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:40:57.676 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:40:57.677 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:40:57.677 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:40:57.677 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:40:58.239 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:40:59.240 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:40:59.240 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:40:59.240 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:40:59.241 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:40:59.241 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:40:59.241 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:40:59.912 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:41:00.913 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:41:00.914 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:41:00.914 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:41:00.914 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:41:00.915 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:41:00.915 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:41:01.325 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3322982530,v1:127.0.0.1:6803/3322982530] [v2:127.0.0.1:6804/3322982530,v1:127.0.0.1:6805/3322982530] exists,up a93d35b5-7d15-49ef-867b-a76ed87911e5 2024-09-16T05:41:01.327 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:41:01.327 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:41:01.327 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:41:01.327 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: get_asok_path osd.0 2024-09-16T05:41:01.327 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:41:01.327 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:41:01.327 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:41:01.327 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:41:01.327 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:41:01.327 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-osd.0.asok 2024-09-16T05:41:01.327 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: CEPH_ARGS= 2024-09-16T05:41:01.327 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: ceph --format=json daemon /tmp/ceph-asok.77187/ceph-osd.0.asok config get osd_max_backfills 2024-09-16T05:41:01.412 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: local 'backfills={"osd_max_backfills":"1"}' 2024-09-16T05:41:01.412 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: test_activate_osd_after_mark_down: echo '{"osd_max_backfills":"1"}' 2024-09-16T05:41:01.413 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: test_activate_osd_after_mark_down: grep --quiet osd_max_backfills 2024-09-16T05:41:01.413 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: test_activate_osd_after_mark_down: kill_daemons td/ceph-helpers TERM osd 2024-09-16T05:41:01.414 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:41:01.414 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:41:01.415 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:41:01.415 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:41:01.415 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:41:01.720 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:41:01.720 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: test_activate_osd_after_mark_down: ceph osd down 0 2024-09-16T05:41:02.104 INFO:tasks.workunit.client.0.smithi178.stderr:osd.0 is already down. 2024-09-16T05:41:02.115 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: test_activate_osd_after_mark_down: wait_for_osd down 0 2024-09-16T05:41:02.116 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2024-09-16T05:41:02.116 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:41:02.116 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:41:02.116 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:41:02.116 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:41:02.116 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:41:02.117 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:41:02.117 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:41:02.117 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2024-09-16T05:41:02.524 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6802/3322982530,v1:127.0.0.1:6803/3322982530] [v2:127.0.0.1:6804/3322982530,v1:127.0.0.1:6805/3322982530] exists a93d35b5-7d15-49ef-867b-a76ed87911e5 2024-09-16T05:41:02.527 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:41:02.527 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:41:02.527 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:41:02.527 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: test_activate_osd_after_mark_down: activate_osd td/ceph-helpers 0 --osd-max-backfills 20 2024-09-16T05:41:02.527 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-09-16T05:41:02.527 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T05:41:02.527 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T05:41:02.528 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T05:41:02.528 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-09-16T05:41:02.528 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T05:41:02.528 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:41:02.528 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:41:02.528 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:41:02.528 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T05:41:02.528 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T05:41:02.528 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T05:41:02.528 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T05:41:02.528 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T05:41:02.528 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T05:41:02.528 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:41:02.529 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:41:02.529 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:41:02.529 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:41:02.529 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:41:02.529 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:41:02.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:41:02.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:41:02.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T05:41:02.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T05:41:02.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:41:02.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:41:02.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:41:02.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:41:02.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T05:41:02.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-max-backfills 20' 2024-09-16T05:41:02.531 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-09-16T05:41:02.531 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T05:41:02.531 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:41:02.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2024-09-16T05:41:02.532 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-09-16T05:41:02.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T05:41:02.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T05:41:02.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T05:41:02.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T05:41:02.553 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:41:02.553+0000 7f4330416640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:41:02.555 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:41:02.555+0000 7f4330416640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:41:02.556 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:41:02.556+0000 7f4330416640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:41:02.943 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T05:41:02.943 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:41:02.943 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:41:02.943 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:41:02.943 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:41:02.944 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:41:02.944 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:41:02.945 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:41:02.945 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:41:02.945 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:41:03.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:41:03.404 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:41:03.404+0000 7f4330416640 -1 Falling back to public interface 2024-09-16T05:41:04.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:41:04.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:41:04.353 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:41:04.354 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:41:04.354 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:41:04.354 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:41:04.730 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:41:05.731 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:41:05.731 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:41:05.732 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:41:05.732 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:41:05.732 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:41:05.732 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:41:06.194 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:41:06.375 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:41:06.374+0000 7f4330416640 -1 osd.0 5 log_to_monitors true 2024-09-16T05:41:07.195 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:41:07.196 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:41:07.196 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:41:07.197 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:41:07.197 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:41:07.197 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:41:07.575 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 8 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6802/1103733493,v1:127.0.0.1:6803/1103733493] [v2:127.0.0.1:6804/1103733493,v1:127.0.0.1:6805/1103733493] exists,up a93d35b5-7d15-49ef-867b-a76ed87911e5 2024-09-16T05:41:07.577 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:41:07.577 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:41:07.577 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:41:07.577 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: get_asok_path osd.0 2024-09-16T05:41:07.577 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:41:07.577 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:41:07.577 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:41:07.577 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:41:07.577 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:41:07.577 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-osd.0.asok 2024-09-16T05:41:07.577 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: CEPH_ARGS= 2024-09-16T05:41:07.577 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: ceph --format=json daemon /tmp/ceph-asok.77187/ceph-osd.0.asok config get osd_max_backfills 2024-09-16T05:41:07.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: local 'backfills={"osd_max_backfills":"20"}' 2024-09-16T05:41:07.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: test_activate_osd_after_mark_down: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2024-09-16T05:41:07.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: test_activate_osd_after_mark_down: teardown td/ceph-helpers 2024-09-16T05:41:07.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:41:07.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:41:07.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:41:07.662 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:41:07.662 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:41:07.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:41:07.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:41:07.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:41:07.767 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:41:07.767 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:41:07.767 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:41:07.768 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:41:07.769 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:41:07.769 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:41:07.769 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:41:07.770 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:41:07.770 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:41:07.770 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:41:07.770 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:41:07.771 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:41:07.772 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:41:07.772 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:41:07.777 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:41:07.777 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:41:07.777 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:41:07.778 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:41:07.779 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:41:07.779 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:41:07.780 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T05:41:07.780 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_activate_osd_skip_benchmark td/ceph-helpers 2024-09-16T05:41:07.780 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:934: test_activate_osd_skip_benchmark: local dir=td/ceph-helpers 2024-09-16T05:41:07.780 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:936: test_activate_osd_skip_benchmark: setup td/ceph-helpers 2024-09-16T05:41:07.780 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T05:41:07.780 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T05:41:07.780 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:41:07.780 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:41:07.780 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:41:07.780 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:41:07.781 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:41:07.781 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:41:07.781 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:41:07.781 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:41:07.781 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:41:07.782 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:41:07.782 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:41:07.782 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:41:07.783 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:41:07.783 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:41:07.783 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:41:07.783 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:41:07.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:41:07.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:41:07.784 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:41:07.785 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:41:07.786 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:41:07.786 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:41:07.787 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:41:07.787 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:41:07.787 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:41:07.787 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:41:07.788 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:41:07.788 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:41:07.788 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T05:41:07.789 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:41:07.789 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:41:07.789 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:41:07.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T05:41:07.793 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:41:07.793 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:41:07.793 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:41:07.793 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T05:41:07.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:938: test_activate_osd_skip_benchmark: run_mon td/ceph-helpers a 2024-09-16T05:41:07.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T05:41:07.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:41:07.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:41:07.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:41:07.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T05:41:07.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-16T05:41:08.192 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:41:08.192 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:41:08.192 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:41:08.192 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:41:08.193 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:41:08.193 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:41:08.193 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:41:08.193 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T05:41:08.384 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:41:08.386 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:41:08.386 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:41:08.387 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:41:08.387 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:41:08.387 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:41:08.387 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:41:08.387 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:41:08.387 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:41:08.387 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:41:08.387 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:41:08.387 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:41:08.387 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:41:08.388 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:41:08.388 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T05:41:08.468 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:41:08.468 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:41:08.468 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:41:08.468 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:41:08.468 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:41:08.469 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:41:08.469 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:41:08.469 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:41:08.469 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:41:08.469 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:41:08.469 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:41:08.469 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:41:08.469 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:41:08.469 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T05:41:08.546 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:939: test_activate_osd_skip_benchmark: run_mgr td/ceph-helpers x 2024-09-16T05:41:08.546 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T05:41:08.547 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T05:41:08.547 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T05:41:08.547 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T05:41:08.547 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T05:41:08.547 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T05:41:09.389 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T05:41:09.390 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:41:09.390 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:41:09.390 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:41:09.390 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:41:09.390 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:41:09.390 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:41:09.390 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:41:09.390 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:41:09.423 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:942: test_activate_osd_skip_benchmark: run_osd td/ceph-helpers 0 --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-09-16T05:41:09.423 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T05:41:09.423 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:41:09.423 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T05:41:09.423 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:41:09.424 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T05:41:09.424 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T05:41:09.424 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:41:09.424 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:41:09.424 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:41:09.424 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T05:41:09.424 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T05:41:09.424 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:41:09.424 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:41:09.424 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T05:41:09.424 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:41:09.424 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:41:09.424 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:41:09.425 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:41:09.425 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:41:09.425 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:41:09.425 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:41:09.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:41:09.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:41:09.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:41:09.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:41:09.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T05:41:09.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T05:41:09.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:41:09.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:41:09.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:41:09.426 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:41:09.426 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:41:09.426 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true' 2024-09-16T05:41:09.426 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T05:41:09.426 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:41:09.426 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f5d0eab8-b608-40b6-ab24-56256ebe3229 2024-09-16T05:41:09.426 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 f5d0eab8-b608-40b6-ab24-56256ebe3229 2024-09-16T05:41:09.427 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f5d0eab8-b608-40b6-ab24-56256ebe3229' 2024-09-16T05:41:09.427 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:41:09.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD1xOdmpFQ2GhAAVGjLYP5c4mYtGSzyvEu8Hg== 2024-09-16T05:41:09.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD1xOdmpFQ2GhAAVGjLYP5c4mYtGSzyvEu8Hg=="}' 2024-09-16T05:41:09.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f5d0eab8-b608-40b6-ab24-56256ebe3229 -i td/ceph-helpers/0/new.json 2024-09-16T05:41:09.709 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:41:09.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T05:41:09.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true --mkfs --key AQD1xOdmpFQ2GhAAVGjLYP5c4mYtGSzyvEu8Hg== --osd-uuid f5d0eab8-b608-40b6-ab24-56256ebe3229 2024-09-16T05:41:09.749 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:41:09.749+0000 7f04ec420640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:41:09.769 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:41:09.769+0000 7f04ec420640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:41:09.770 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:41:09.770+0000 7f04ec420640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:41:09.770 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:41:09.770+0000 7f04ec420640 -1 bdev(0x55e26e8e5c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-16T05:41:09.771 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:41:09.770+0000 7f04ec420640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T05:41:15.219 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T05:41:15.219 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:41:15.220 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T05:41:15.220 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T05:41:15.221 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:41:15.738 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:41:15.739 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T05:41:15.739 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-09-16T05:41:15.739 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:41:15.739 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:41:15.739 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:41:15.766 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:41:15.765+0000 7f4430bdb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:41:15.784 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:41:15.783+0000 7f4430bdb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:41:15.785 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:41:15.785+0000 7f4430bdb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:41:16.173 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T05:41:16.173 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:41:16.173 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:41:16.173 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:41:16.173 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:41:16.173 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:41:16.173 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:41:16.174 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:41:16.174 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:41:16.174 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:41:16.544 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:41:17.396 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:41:17.396+0000 7f4430bdb640 -1 Falling back to public interface 2024-09-16T05:41:17.545 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:41:17.545 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:41:17.546 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:41:17.546 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:41:17.546 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:41:17.546 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:41:17.965 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:41:18.967 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:41:18.967 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:41:18.968 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:41:18.969 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:41:18.969 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:41:18.969 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:41:19.351 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:41:20.063 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:41:20.062+0000 7f4430bdb640 -1 osd.0 0 log_to_monitors true 2024-09-16T05:41:20.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:41:20.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:41:20.353 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:41:20.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:41:20.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:41:20.354 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:41:20.915 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3555083773,v1:127.0.0.1:6803/3555083773] [v2:127.0.0.1:6804/3555083773,v1:127.0.0.1:6805/3555083773] exists,up f5d0eab8-b608-40b6-ab24-56256ebe3229 2024-09-16T05:41:20.918 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:41:20.918 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:41:20.918 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:41:20.918 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-09-16T05:41:20.918 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:41:20.919 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:41:20.919 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:41:20.919 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:41:20.919 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:41:20.920 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-osd.0.asok 2024-09-16T05:41:20.920 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-09-16T05:41:20.920 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.77187/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_hdd 2024-09-16T05:41:20.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: local 'max_iops_hdd_def={"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2024-09-16T05:41:21.001 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-09-16T05:41:21.001 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:41:21.001 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:41:21.001 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:41:21.002 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:41:21.002 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:41:21.002 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-osd.0.asok 2024-09-16T05:41:21.002 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-09-16T05:41:21.002 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.77187/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_ssd 2024-09-16T05:41:21.085 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: local 'max_iops_ssd_def={"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2024-09-16T05:41:21.085 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:949: test_activate_osd_skip_benchmark: kill_daemons td/ceph-helpers TERM osd 2024-09-16T05:41:21.085 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:41:21.085 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:41:21.085 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:41:21.086 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:41:21.086 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:41:21.391 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:41:21.391 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:950: test_activate_osd_skip_benchmark: ceph osd down 0 2024-09-16T05:41:21.807 INFO:tasks.workunit.client.0.smithi178.stderr:osd.0 is already down. 2024-09-16T05:41:21.818 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:951: test_activate_osd_skip_benchmark: wait_for_osd down 0 2024-09-16T05:41:21.818 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2024-09-16T05:41:21.818 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:41:21.818 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:41:21.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:41:21.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:41:21.819 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:41:21.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:41:21.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:41:21.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2024-09-16T05:41:22.228 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6802/3555083773,v1:127.0.0.1:6803/3555083773] [v2:127.0.0.1:6804/3555083773,v1:127.0.0.1:6805/3555083773] exists f5d0eab8-b608-40b6-ab24-56256ebe3229 2024-09-16T05:41:22.232 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:41:22.233 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:41:22.233 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:41:22.233 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:955: test_activate_osd_skip_benchmark: activate_osd td/ceph-helpers 0 --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-09-16T05:41:22.233 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-09-16T05:41:22.233 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T05:41:22.233 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T05:41:22.233 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T05:41:22.233 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-09-16T05:41:22.233 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T05:41:22.233 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:41:22.233 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:41:22.234 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:41:22.234 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T05:41:22.234 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T05:41:22.234 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T05:41:22.234 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T05:41:22.234 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T05:41:22.234 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T05:41:22.234 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:41:22.234 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:41:22.234 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:41:22.235 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:41:22.235 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:41:22.235 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:41:22.235 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:41:22.235 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:41:22.235 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T05:41:22.235 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T05:41:22.235 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:41:22.235 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:41:22.235 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:41:22.235 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:41:22.235 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T05:41:22.235 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true' 2024-09-16T05:41:22.235 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-09-16T05:41:22.235 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T05:41:22.236 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:41:22.236 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-09-16T05:41:22.237 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-09-16T05:41:22.237 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T05:41:22.237 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T05:41:22.237 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T05:41:22.237 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T05:41:22.259 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:41:22.258+0000 7f712ee3e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:41:22.260 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:41:22.260+0000 7f712ee3e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:41:22.262 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:41:22.261+0000 7f712ee3e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:41:22.646 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T05:41:22.646 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:41:22.646 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:41:22.646 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:41:22.646 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:41:22.646 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:41:22.646 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:41:22.647 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:41:22.647 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:41:22.647 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:41:23.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:41:23.596 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:41:23.596+0000 7f712ee3e640 -1 Falling back to public interface 2024-09-16T05:41:24.056 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:41:24.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:41:24.057 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:41:24.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:41:24.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:41:24.058 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:41:24.473 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:41:25.474 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:41:25.474 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:41:25.474 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:41:25.475 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:41:25.475 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:41:25.475 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:41:25.855 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:41:26.707 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:41:26.706+0000 7f712ee3e640 -1 osd.0 5 log_to_monitors true 2024-09-16T05:41:26.856 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:41:26.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:41:26.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:41:26.857 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:41:26.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:41:26.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:41:27.291 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 8 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6802/1275629280,v1:127.0.0.1:6803/1275629280] [v2:127.0.0.1:6804/1275629280,v1:127.0.0.1:6805/1275629280] exists,up f5d0eab8-b608-40b6-ab24-56256ebe3229 2024-09-16T05:41:27.292 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:41:27.292 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:41:27.292 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:41:27.292 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-09-16T05:41:27.292 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:41:27.293 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:41:27.293 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:41:27.293 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:41:27.293 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:41:27.293 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-osd.0.asok 2024-09-16T05:41:27.293 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-09-16T05:41:27.293 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.77187/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_hdd 2024-09-16T05:41:27.377 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: local 'max_iops_hdd_after_boot={"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2024-09-16T05:41:27.378 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-09-16T05:41:27.378 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:41:27.379 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:41:27.379 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:41:27.379 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:41:27.379 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:41:27.379 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-osd.0.asok 2024-09-16T05:41:27.379 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-09-16T05:41:27.379 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.77187/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_ssd 2024-09-16T05:41:27.463 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: local 'max_iops_ssd_after_boot={"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2024-09-16T05:41:27.463 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: test_activate_osd_skip_benchmark: test '{"osd_mclock_max_capacity_iops_hdd":"315.000000"}' = '{"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2024-09-16T05:41:27.464 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: test_activate_osd_skip_benchmark: test '{"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' = '{"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2024-09-16T05:41:27.464 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: test_activate_osd_skip_benchmark: teardown td/ceph-helpers 2024-09-16T05:41:27.464 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:41:27.464 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:41:27.464 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:41:27.464 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:41:27.464 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:41:27.464 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:41:27.464 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:41:27.464 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:41:27.569 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:41:27.569 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:41:27.570 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:41:27.570 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:41:27.571 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:41:27.571 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:41:27.571 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:41:27.572 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:41:27.572 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:41:27.572 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:41:27.572 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:41:27.573 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:41:27.574 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:41:27.574 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:41:27.577 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:41:27.577 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:41:27.577 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:41:27.577 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:41:27.579 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:41:27.579 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:41:27.579 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T05:41:27.579 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_destroy_osd td/ceph-helpers 2024-09-16T05:41:27.579 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:796: test_destroy_osd: local dir=td/ceph-helpers 2024-09-16T05:41:27.580 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:798: test_destroy_osd: setup td/ceph-helpers 2024-09-16T05:41:27.580 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T05:41:27.580 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T05:41:27.580 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:41:27.580 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:41:27.580 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:41:27.580 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:41:27.580 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:41:27.580 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:41:27.580 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:41:27.580 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:41:27.581 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:41:27.581 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:41:27.581 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:41:27.582 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:41:27.582 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:41:27.582 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:41:27.582 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:41:27.583 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:41:27.583 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:41:27.583 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:41:27.584 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:41:27.584 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:41:27.585 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:41:27.585 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:41:27.586 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:41:27.586 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:41:27.586 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:41:27.586 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:41:27.587 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:41:27.587 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:41:27.587 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T05:41:27.588 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:41:27.589 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:41:27.589 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:41:27.589 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T05:41:27.591 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:41:27.591 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:41:27.591 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:41:27.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T05:41:27.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:799: test_destroy_osd: run_mon td/ceph-helpers a 2024-09-16T05:41:27.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T05:41:27.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:41:27.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:41:27.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:41:27.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T05:41:27.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-16T05:41:27.967 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:41:27.967 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:41:27.967 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:41:27.967 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:41:27.967 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:41:27.967 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:41:27.967 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:41:27.967 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T05:41:28.150 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:41:28.151 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:41:28.151 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:41:28.151 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:41:28.152 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:41:28.153 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:41:28.153 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:41:28.153 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:41:28.153 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:41:28.153 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:41:28.154 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:41:28.154 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:41:28.154 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:41:28.154 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:41:28.154 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T05:41:28.232 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:41:28.232 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:41:28.232 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:41:28.232 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:41:28.233 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:41:28.233 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:41:28.233 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:41:28.233 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:41:28.233 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:41:28.233 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:41:28.233 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:41:28.233 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:41:28.233 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:41:28.233 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T05:41:28.309 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:800: test_destroy_osd: run_mgr td/ceph-helpers x 2024-09-16T05:41:28.310 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T05:41:28.310 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T05:41:28.310 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T05:41:28.310 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T05:41:28.310 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T05:41:28.310 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T05:41:29.189 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T05:41:29.189 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:41:29.189 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:41:29.189 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:41:29.189 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:41:29.189 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:41:29.189 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:41:29.189 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:41:29.190 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:41:29.220 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:801: test_destroy_osd: run_osd td/ceph-helpers 0 2024-09-16T05:41:29.220 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T05:41:29.220 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:41:29.220 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T05:41:29.220 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:41:29.220 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T05:41:29.221 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T05:41:29.221 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:41:29.221 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:41:29.221 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:41:29.221 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T05:41:29.221 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T05:41:29.221 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:41:29.221 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:41:29.221 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T05:41:29.221 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:41:29.221 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:41:29.221 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:41:29.221 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:41:29.221 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:41:29.222 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:41:29.222 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:41:29.222 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:41:29.222 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:41:29.222 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:41:29.222 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:41:29.222 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T05:41:29.222 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T05:41:29.222 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:41:29.222 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:41:29.222 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:41:29.222 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:41:29.222 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:41:29.222 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:41:29.222 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T05:41:29.223 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:41:29.223 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0d981323-dc44-43dd-9760-e5f8e1dbf005 2024-09-16T05:41:29.223 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0d981323-dc44-43dd-9760-e5f8e1dbf005' 2024-09-16T05:41:29.223 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 0d981323-dc44-43dd-9760-e5f8e1dbf005 2024-09-16T05:41:29.224 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:41:29.240 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAJxedmFPpBDhAAfG07V44CqXO0o48PWb/lUg== 2024-09-16T05:41:29.240 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAJxedmFPpBDhAAfG07V44CqXO0o48PWb/lUg=="}' 2024-09-16T05:41:29.240 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0d981323-dc44-43dd-9760-e5f8e1dbf005 -i td/ceph-helpers/0/new.json 2024-09-16T05:41:29.508 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:41:29.518 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T05:41:29.519 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAJxedmFPpBDhAAfG07V44CqXO0o48PWb/lUg== --osd-uuid 0d981323-dc44-43dd-9760-e5f8e1dbf005 2024-09-16T05:41:29.548 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:41:29.547+0000 7f57ccc1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:41:29.577 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:41:29.577+0000 7f57ccc1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:41:29.578 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:41:29.578+0000 7f57ccc1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:41:29.579 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:41:29.578+0000 7f57ccc1d640 -1 bdev(0x5648ab5efc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-16T05:41:29.579 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:41:29.578+0000 7f57ccc1d640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T05:41:34.829 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T05:41:34.829 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:41:34.830 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T05:41:34.831 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T05:41:34.831 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:41:35.348 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:41:35.349 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T05:41:35.349 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:41:35.349 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:41:35.349 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:41:35.349 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:41:35.375 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:41:35.375+0000 7fa7a48f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:41:35.377 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:41:35.377+0000 7fa7a48f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:41:35.378 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:41:35.378+0000 7fa7a48f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:41:35.723 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T05:41:35.724 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:41:35.724 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:41:35.724 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:41:35.724 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:41:35.724 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:41:35.724 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:41:35.725 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:41:35.725 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:41:35.725 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:41:36.097 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:41:36.474 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:41:36.474+0000 7fa7a48f1640 -1 Falling back to public interface 2024-09-16T05:41:37.099 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:41:37.099 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:41:37.099 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:41:37.099 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:41:37.100 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:41:37.100 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:41:37.519 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:41:38.522 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:41:38.522 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:41:38.523 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:41:38.524 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:41:38.524 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:41:38.525 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:41:38.901 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:41:39.715 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:41:39.713+0000 7fa7a48f1640 -1 osd.0 0 log_to_monitors true 2024-09-16T05:41:39.903 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:41:39.903 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:41:39.903 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:41:39.903 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:41:39.904 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:41:39.904 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:41:40.292 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:41:41.293 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:41:41.294 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:41:41.294 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:41:41.294 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:41:41.294 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:41:41.295 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:41:41.787 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:41:42.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:41:42.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:41:42.789 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:41:42.790 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:41:42.790 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:41:42.790 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:41:43.170 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:41:44.172 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:41:44.172 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:41:44.172 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:41:44.173 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:41:44.173 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:41:44.173 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:41:44.556 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:41:45.558 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:41:45.558 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:41:45.558 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:41:45.559 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:41:45.559 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:41:45.559 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:41:45.965 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:41:46.966 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:41:46.966 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:41:46.967 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:41:46.967 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:41:46.967 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:41:46.967 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:41:47.488 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:41:48.490 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:41:48.490 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:41:48.490 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:41:48.491 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:41:48.491 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:41:48.491 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:41:48.915 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:41:49.916 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:41:49.916 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:41:49.917 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:41:49.917 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:41:49.917 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:41:49.917 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:41:50.447 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:41:51.449 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:41:51.449 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:41:51.449 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:41:51.450 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:41:51.450 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:41:51.450 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:41:52.308 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:41:53.310 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:41:53.310 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:41:53.310 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:41:53.310 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:41:53.310 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:41:53.311 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:41:54.463 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:41:55.464 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:41:55.464 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:41:55.464 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:41:55.465 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:41:55.465 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:41:55.465 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:41:56.205 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:41:57.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:41:57.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:41:57.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:41:57.208 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:41:57.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:41:57.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:41:57.925 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:41:58.926 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:41:58.926 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:41:58.927 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:41:58.928 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:41:58.928 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:41:58.928 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:41:59.588 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:42:00.589 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:42:00.590 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:42:00.590 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:42:00.590 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:42:00.590 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:42:00.591 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:42:01.261 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:42:02.262 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:42:02.263 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:42:02.263 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:42:02.263 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:42:02.263 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:42:02.263 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:42:02.683 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3839417200,v1:127.0.0.1:6803/3839417200] [v2:127.0.0.1:6804/3839417200,v1:127.0.0.1:6805/3839417200] exists,up 0d981323-dc44-43dd-9760-e5f8e1dbf005 2024-09-16T05:42:02.684 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:42:02.684 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:42:02.684 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:42:02.684 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:802: test_destroy_osd: destroy_osd td/ceph-helpers 0 2024-09-16T05:42:02.684 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:784: destroy_osd: local dir=td/ceph-helpers 2024-09-16T05:42:02.685 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:785: destroy_osd: local id=0 2024-09-16T05:42:02.685 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:787: destroy_osd: ceph osd out osd.0 2024-09-16T05:42:03.285 INFO:tasks.workunit.client.0.smithi178.stderr:osd.0 is already out. 2024-09-16T05:42:03.296 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:788: destroy_osd: kill_daemons td/ceph-helpers TERM osd.0 2024-09-16T05:42:03.297 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:42:03.297 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:42:03.297 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:42:03.297 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:42:03.297 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:42:03.602 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:42:03.602 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:789: destroy_osd: ceph osd down osd.0 2024-09-16T05:42:04.002 INFO:tasks.workunit.client.0.smithi178.stderr:osd.0 is already down. 2024-09-16T05:42:04.013 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:790: destroy_osd: ceph osd purge osd.0 --yes-i-really-mean-it 2024-09-16T05:42:04.436 INFO:tasks.workunit.client.0.smithi178.stderr:osd.0 does not exist 2024-09-16T05:42:04.448 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:791: destroy_osd: teardown td/ceph-helpers/0 2024-09-16T05:42:04.448 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers/0 2024-09-16T05:42:04.448 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:42:04.449 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers/0 KILL 2024-09-16T05:42:04.449 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:42:04.449 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:42:04.449 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:42:04.449 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:42:04.449 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:42:04.450 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:42:04.450 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:42:04.450 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:42:04.451 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:42:04.452 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:42:04.452 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:42:04.452 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:42:04.453 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:42:04.453 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:42:04.454 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:42:04.454 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:42:04.454 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:42:04.456 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:42:04.457 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers/0 2024-09-16T05:42:04.459 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:42:04.459 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:42:04.459 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:42:04.459 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:42:04.460 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:42:04.460 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:42:04.460 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:792: destroy_osd: rm -fr td/ceph-helpers/0 2024-09-16T05:42:04.461 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:803: test_destroy_osd: ceph osd dump 2024-09-16T05:42:04.461 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:803: test_destroy_osd: grep 'osd. ' 2024-09-16T05:42:04.829 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:804: test_destroy_osd: teardown td/ceph-helpers 2024-09-16T05:42:04.829 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:42:04.829 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:42:04.829 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:42:04.829 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:42:04.829 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:42:04.829 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:42:04.829 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:42:04.829 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:42:04.934 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:42:04.935 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:42:04.936 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:42:04.936 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:42:04.937 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:42:04.937 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:42:04.937 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:42:04.937 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:42:04.937 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:42:04.938 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:42:04.938 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:42:04.938 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:42:04.939 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:42:04.940 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:42:04.943 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:42:04.943 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:42:04.943 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:42:04.943 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:42:04.945 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:42:04.945 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:42:04.945 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T05:42:04.945 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_display_logs td/ceph-helpers 2024-09-16T05:42:04.945 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2102: test_display_logs: local dir=td/ceph-helpers 2024-09-16T05:42:04.945 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2104: test_display_logs: setup td/ceph-helpers 2024-09-16T05:42:04.945 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T05:42:04.945 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T05:42:04.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:42:04.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:42:04.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:42:04.946 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:42:04.946 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:42:04.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:42:04.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:42:04.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:42:04.947 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:42:04.947 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:42:04.947 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:42:04.947 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:42:04.948 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:42:04.948 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:42:04.948 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:42:04.949 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:42:04.949 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:42:04.949 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:42:04.950 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:42:04.950 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:42:04.951 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:42:04.951 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:42:04.952 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:42:04.952 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:42:04.952 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:42:04.952 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:42:04.953 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:42:04.953 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:42:04.953 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T05:42:04.954 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:42:04.954 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:42:04.954 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:42:04.955 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T05:42:04.956 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:42:04.956 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:42:04.956 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:42:04.956 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T05:42:04.957 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2105: test_display_logs: run_mon td/ceph-helpers a 2024-09-16T05:42:04.957 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T05:42:04.957 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:42:04.957 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:42:04.957 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:42:04.957 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T05:42:04.957 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-16T05:42:05.343 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:42:05.343 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:42:05.343 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:42:05.343 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:42:05.343 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:42:05.344 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:42:05.344 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:42:05.344 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T05:42:05.510 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:42:05.512 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:42:05.512 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:42:05.512 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:42:05.512 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:42:05.512 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:42:05.512 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:42:05.512 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:42:05.512 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:42:05.512 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:42:05.512 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:42:05.512 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:42:05.512 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:42:05.513 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:42:05.513 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T05:42:05.594 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:42:05.594 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:42:05.594 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:42:05.594 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:42:05.594 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:42:05.595 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:42:05.595 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:42:05.595 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:42:05.595 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:42:05.595 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:42:05.595 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:42:05.595 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:42:05.595 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:42:05.595 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T05:42:05.673 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2106: test_display_logs: kill_daemons td/ceph-helpers 2024-09-16T05:42:05.673 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:42:05.673 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:42:05.673 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:42:05.673 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:42:05.673 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:42:06.980 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:42:06.980 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2107: test_display_logs: display_logs td/ceph-helpers 2024-09-16T05:42:06.980 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2092: display_logs: local dir=td/ceph-helpers 2024-09-16T05:42:06.980 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2094: display_logs: find td/ceph-helpers -maxdepth 1 -name '*.log' 2024-09-16T05:42:06.980 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2095: display_logs: read file 2024-09-16T05:42:06.981 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2096: display_logs: echo '======================= td/ceph-helpers/mon.a.log' 2024-09-16T05:42:06.981 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2097: display_logs: cat td/ceph-helpers/mon.a.log 2024-09-16T05:42:06.982 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2095: display_logs: read file 2024-09-16T05:42:06.982 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2108: test_display_logs: grep --quiet mon.a.log td/ceph-helpers/log.out 2024-09-16T05:42:06.984 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2109: test_display_logs: teardown td/ceph-helpers 2024-09-16T05:42:06.985 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:42:06.985 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:42:06.985 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:42:06.985 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:42:06.985 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:42:06.985 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:42:06.985 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:42:06.985 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:42:06.986 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:42:06.987 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:42:06.988 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:42:06.988 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:42:06.989 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:42:06.989 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:42:06.990 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:42:06.991 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:42:06.991 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:42:06.991 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:42:06.991 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:42:06.992 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:42:06.993 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:42:06.993 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:42:06.995 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:42:06.995 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:42:06.995 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:42:06.995 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:42:06.997 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:42:06.997 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:42:06.997 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T05:42:06.997 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_erasure_code_plugin_exists td/ceph-helpers 2024-09-16T05:42:06.997 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: test_erasure_code_plugin_exists: local dir=td/ceph-helpers 2024-09-16T05:42:06.997 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: test_erasure_code_plugin_exists: setup td/ceph-helpers 2024-09-16T05:42:06.998 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T05:42:06.998 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T05:42:06.998 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:42:06.998 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:42:06.998 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:42:06.998 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:42:06.998 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:42:06.998 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:42:06.998 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:42:06.998 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:42:06.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:42:06.999 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:42:07.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:42:07.000 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:42:07.001 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:42:07.001 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:42:07.002 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:42:07.002 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:42:07.003 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:42:07.003 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:42:07.003 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:42:07.004 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:42:07.005 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:42:07.005 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:42:07.007 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:42:07.007 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:42:07.007 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:42:07.007 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:42:07.008 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:42:07.008 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:42:07.008 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T05:42:07.009 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:42:07.009 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:42:07.009 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:42:07.009 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T05:42:07.012 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:42:07.012 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:42:07.012 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:42:07.012 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T05:42:07.012 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: test_erasure_code_plugin_exists: run_mon td/ceph-helpers a 2024-09-16T05:42:07.012 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T05:42:07.012 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:42:07.012 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:42:07.012 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:42:07.012 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T05:42:07.012 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-16T05:42:07.391 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:42:07.391 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:42:07.391 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:42:07.391 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:42:07.391 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:42:07.391 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:42:07.391 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:42:07.391 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T05:42:07.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:42:07.593 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:42:07.593 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:42:07.593 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:42:07.593 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:42:07.594 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:42:07.595 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:42:07.595 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:42:07.595 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:42:07.595 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:42:07.595 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:42:07.595 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:42:07.595 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:42:07.595 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:42:07.595 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T05:42:07.685 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:42:07.685 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:42:07.685 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:42:07.685 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:42:07.685 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:42:07.685 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:42:07.686 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:42:07.686 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:42:07.686 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:42:07.686 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:42:07.686 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:42:07.686 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:42:07.686 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:42:07.686 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T05:42:07.766 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: test_erasure_code_plugin_exists: run_mgr td/ceph-helpers x 2024-09-16T05:42:07.766 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T05:42:07.766 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T05:42:07.766 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T05:42:07.766 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T05:42:07.766 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T05:42:07.767 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T05:42:08.612 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T05:42:08.612 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:42:08.613 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:42:08.613 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:42:08.613 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:42:08.613 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:42:08.613 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:42:08.613 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:42:08.613 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:42:08.641 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: test_erasure_code_plugin_exists: erasure_code_plugin_exists jerasure 2024-09-16T05:42:08.641 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2051: erasure_code_plugin_exists: local plugin=jerasure 2024-09-16T05:42:08.641 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2052: erasure_code_plugin_exists: local status 2024-09-16T05:42:08.641 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2053: erasure_code_plugin_exists: local grepstr 2024-09-16T05:42:08.642 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2054: erasure_code_plugin_exists: local s 2024-09-16T05:42:08.642 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2055: erasure_code_plugin_exists: case `uname` in 2024-09-16T05:42:08.642 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2055: erasure_code_plugin_exists: uname 2024-09-16T05:42:08.642 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2057: erasure_code_plugin_exists: grepstr='jerasure.*No such file' 2024-09-16T05:42:08.642 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2060: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=jerasure 2024-09-16T05:42:08.926 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2060: erasure_code_plugin_exists: s= 2024-09-16T05:42:08.926 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2061: erasure_code_plugin_exists: local status=0 2024-09-16T05:42:08.926 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2062: erasure_code_plugin_exists: '[' 0 -eq 0 ']' 2024-09-16T05:42:08.926 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: erasure_code_plugin_exists: ceph osd erasure-code-profile rm TESTPROFILE 2024-09-16T05:42:09.233 INFO:tasks.workunit.client.0.smithi178.stderr:erasure-code-profile TESTPROFILE does not exist 2024-09-16T05:42:09.243 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: erasure_code_plugin_exists: return 0 2024-09-16T05:42:09.243 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: test_erasure_code_plugin_exists: erasure_code_plugin_exists FAKE 2024-09-16T05:42:09.244 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2051: erasure_code_plugin_exists: local plugin=FAKE 2024-09-16T05:42:09.244 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2052: erasure_code_plugin_exists: local status 2024-09-16T05:42:09.244 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2053: erasure_code_plugin_exists: local grepstr 2024-09-16T05:42:09.244 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2054: erasure_code_plugin_exists: local s 2024-09-16T05:42:09.244 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2055: erasure_code_plugin_exists: case `uname` in 2024-09-16T05:42:09.244 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2055: erasure_code_plugin_exists: uname 2024-09-16T05:42:09.244 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2057: erasure_code_plugin_exists: grepstr='FAKE.*No such file' 2024-09-16T05:42:09.245 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2060: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=FAKE 2024-09-16T05:42:09.402 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2060: erasure_code_plugin_exists: s='Error EIO: load dlopen(/usr/lib64/ceph/erasure-code/libec_FAKE.so): /usr/lib64/ceph/erasure-code/libec_FAKE.so: cannot open shared object file: No such file or directory' 2024-09-16T05:42:09.402 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2061: erasure_code_plugin_exists: local status=5 2024-09-16T05:42:09.402 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2062: erasure_code_plugin_exists: '[' 5 -eq 0 ']' 2024-09-16T05:42:09.402 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: erasure_code_plugin_exists: echo Error EIO: load 'dlopen(/usr/lib64/ceph/erasure-code/libec_FAKE.so):' /usr/lib64/ceph/erasure-code/libec_FAKE.so: cannot open shared object file: No such file or directory 2024-09-16T05:42:09.402 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: erasure_code_plugin_exists: grep --quiet 'FAKE.*No such file' 2024-09-16T05:42:09.403 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: erasure_code_plugin_exists: return 5 2024-09-16T05:42:09.403 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: test_erasure_code_plugin_exists: teardown td/ceph-helpers 2024-09-16T05:42:09.403 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:42:09.404 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:42:09.404 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:42:09.404 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:42:09.404 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:42:09.404 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:42:09.404 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:42:09.404 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:42:09.508 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:42:09.508 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:42:09.509 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:42:09.509 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:42:09.510 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:42:09.510 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:42:09.510 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:42:09.511 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:42:09.511 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:42:09.512 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:42:09.512 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:42:09.512 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:42:09.513 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:42:09.513 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:42:09.515 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:42:09.515 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:42:09.515 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:42:09.515 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:42:09.518 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:42:09.519 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:42:09.519 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T05:42:09.519 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_expect_failure td/ceph-helpers 2024-09-16T05:42:09.519 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: test_expect_failure: local dir=td/ceph-helpers 2024-09-16T05:42:09.519 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: test_expect_failure: setup td/ceph-helpers 2024-09-16T05:42:09.519 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T05:42:09.519 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T05:42:09.519 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:42:09.519 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:42:09.519 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:42:09.519 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:42:09.519 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:42:09.519 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:42:09.519 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:42:09.520 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:42:09.520 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:42:09.520 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:42:09.520 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:42:09.521 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:42:09.521 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:42:09.521 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:42:09.521 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:42:09.521 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:42:09.522 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:42:09.522 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:42:09.522 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:42:09.522 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:42:09.523 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:42:09.523 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:42:09.524 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:42:09.524 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:42:09.524 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:42:09.524 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:42:09.525 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:42:09.525 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:42:09.525 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T05:42:09.526 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:42:09.526 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:42:09.526 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:42:09.527 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T05:42:09.528 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:42:09.528 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:42:09.528 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:42:09.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T05:42:09.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo FAIL ; exit 1' 2024-09-16T05:42:09.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1962: expect_failure: local dir=td/ceph-helpers 2024-09-16T05:42:09.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1963: expect_failure: shift 2024-09-16T05:42:09.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1964: expect_failure: local expected=FAIL 2024-09-16T05:42:09.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1965: expect_failure: shift 2024-09-16T05:42:09.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1966: expect_failure: local success 2024-09-16T05:42:09.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: expect_failure: bash -c 'echo FAIL ; exit 1' 2024-09-16T05:42:09.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: success=false 2024-09-16T05:42:09.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: false 2024-09-16T05:42:09.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-09-16T05:42:09.531 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: return 0 2024-09-16T05:42:09.531 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo FAIL ; exit 0' 2024-09-16T05:42:09.531 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1962: expect_failure: local dir=td/ceph-helpers 2024-09-16T05:42:09.531 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1963: expect_failure: shift 2024-09-16T05:42:09.531 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1964: expect_failure: local expected=FAIL 2024-09-16T05:42:09.531 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1965: expect_failure: shift 2024-09-16T05:42:09.531 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1966: expect_failure: local success 2024-09-16T05:42:09.531 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: expect_failure: bash -c 'echo FAIL ; exit 0' 2024-09-16T05:42:09.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: expect_failure: success=true 2024-09-16T05:42:09.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: true 2024-09-16T05:42:09.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1975: expect_failure: cat td/ceph-helpers/out 2024-09-16T05:42:09.532 INFO:tasks.workunit.client.0.smithi178.stderr:FAIL 2024-09-16T05:42:09.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1976: expect_failure: return 1 2024-09-16T05:42:09.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: test_expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-09-16T05:42:09.533 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo UNEXPECTED ; exit 1' 2024-09-16T05:42:09.534 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1962: expect_failure: local dir=td/ceph-helpers 2024-09-16T05:42:09.534 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1963: expect_failure: shift 2024-09-16T05:42:09.534 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1964: expect_failure: local expected=FAIL 2024-09-16T05:42:09.534 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1965: expect_failure: shift 2024-09-16T05:42:09.534 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1966: expect_failure: local success 2024-09-16T05:42:09.534 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: expect_failure: bash -c 'echo UNEXPECTED ; exit 1' 2024-09-16T05:42:09.535 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: success=false 2024-09-16T05:42:09.535 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: false 2024-09-16T05:42:09.535 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-09-16T05:42:09.535 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1975: expect_failure: cat td/ceph-helpers/out 2024-09-16T05:42:09.536 INFO:tasks.workunit.client.0.smithi178.stderr:UNEXPECTED 2024-09-16T05:42:09.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1976: expect_failure: return 1 2024-09-16T05:42:09.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: test_expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-09-16T05:42:09.537 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1993: test_expect_failure: teardown td/ceph-helpers 2024-09-16T05:42:09.538 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:42:09.538 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:42:09.538 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:42:09.538 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:42:09.538 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:42:09.538 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:42:09.538 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:42:09.538 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:42:09.539 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:42:09.539 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:42:09.540 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:42:09.540 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:42:09.541 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:42:09.541 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:42:09.541 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:42:09.542 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:42:09.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:42:09.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:42:09.543 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:42:09.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:42:09.544 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:42:09.544 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:42:09.545 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:42:09.546 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:42:09.546 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:42:09.546 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:42:09.548 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:42:09.548 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:42:09.548 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T05:42:09.548 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_flush_pg_stats td/ceph-helpers 2024-09-16T05:42:09.548 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2225: test_flush_pg_stats: local dir=td/ceph-helpers 2024-09-16T05:42:09.548 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: test_flush_pg_stats: setup td/ceph-helpers 2024-09-16T05:42:09.548 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T05:42:09.548 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T05:42:09.548 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:42:09.548 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:42:09.549 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:42:09.549 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:42:09.549 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:42:09.549 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:42:09.549 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:42:09.549 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:42:09.550 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:42:09.550 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:42:09.550 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:42:09.550 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:42:09.551 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:42:09.551 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:42:09.551 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:42:09.552 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:42:09.552 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:42:09.552 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:42:09.552 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:42:09.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:42:09.554 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:42:09.554 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:42:09.555 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:42:09.555 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:42:09.555 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:42:09.555 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:42:09.556 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:42:09.556 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:42:09.556 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T05:42:09.557 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:42:09.557 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:42:09.557 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:42:09.557 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T05:42:09.559 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:42:09.559 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:42:09.559 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:42:09.559 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T05:42:09.559 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: test_flush_pg_stats: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:42:09.560 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T05:42:09.560 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:42:09.560 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:42:09.560 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:42:09.560 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T05:42:09.560 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:42:09.942 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:42:09.942 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:42:09.943 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:42:09.943 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:42:09.943 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:42:09.943 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:42:09.943 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:42:09.943 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:42:10.184 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:42:10.185 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:42:10.185 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:42:10.186 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:42:10.186 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:42:10.186 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:42:10.186 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:42:10.186 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:42:10.186 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:42:10.186 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:42:10.186 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:42:10.186 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:42:10.186 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:42:10.187 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:42:10.187 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T05:42:10.267 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:42:10.268 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:42:10.268 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:42:10.268 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:42:10.268 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:42:10.268 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:42:10.268 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:42:10.268 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:42:10.268 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:42:10.268 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:42:10.268 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:42:10.268 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:42:10.268 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:42:10.268 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T05:42:10.345 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: test_flush_pg_stats: run_mgr td/ceph-helpers x 2024-09-16T05:42:10.345 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T05:42:10.345 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T05:42:10.345 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T05:42:10.345 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T05:42:10.345 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T05:42:10.346 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T05:42:11.198 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T05:42:11.198 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:42:11.198 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:42:11.199 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:42:11.199 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:42:11.199 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:42:11.199 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:42:11.199 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:42:11.199 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:42:11.231 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: test_flush_pg_stats: run_osd td/ceph-helpers 0 2024-09-16T05:42:11.231 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T05:42:11.231 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:42:11.231 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T05:42:11.231 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:42:11.232 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T05:42:11.232 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T05:42:11.232 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:42:11.232 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:42:11.232 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:42:11.232 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T05:42:11.232 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T05:42:11.232 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:42:11.232 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:42:11.232 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T05:42:11.232 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:42:11.232 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:42:11.232 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:42:11.233 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:42:11.233 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:42:11.233 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:42:11.233 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:42:11.233 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:42:11.233 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:42:11.233 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:42:11.234 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:42:11.234 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T05:42:11.234 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T05:42:11.234 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:42:11.234 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:42:11.234 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:42:11.234 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:42:11.234 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:42:11.234 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:42:11.234 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T05:42:11.234 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:42:11.234 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2227ad22-d227-4b1d-b4ba-7fa796ce86e3 2024-09-16T05:42:11.235 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2227ad22-d227-4b1d-b4ba-7fa796ce86e3' 2024-09-16T05:42:11.235 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 2227ad22-d227-4b1d-b4ba-7fa796ce86e3 2024-09-16T05:42:11.235 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:42:11.251 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAzxedm6cHcDhAAwlCtCHoCJv2D9MUDfdDERA== 2024-09-16T05:42:11.251 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAzxedm6cHcDhAAwlCtCHoCJv2D9MUDfdDERA=="}' 2024-09-16T05:42:11.251 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2227ad22-d227-4b1d-b4ba-7fa796ce86e3 -i td/ceph-helpers/0/new.json 2024-09-16T05:42:11.525 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:42:11.534 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T05:42:11.535 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAzxedm6cHcDhAAwlCtCHoCJv2D9MUDfdDERA== --osd-uuid 2227ad22-d227-4b1d-b4ba-7fa796ce86e3 2024-09-16T05:42:11.565 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:42:11.564+0000 7f4bcd21c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:42:11.585 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:42:11.584+0000 7f4bcd21c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:42:11.586 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:42:11.586+0000 7f4bcd21c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:42:11.586 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:42:11.586+0000 7f4bcd21c640 -1 bdev(0x55dfa8bffc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-16T05:42:11.586 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:42:11.586+0000 7f4bcd21c640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T05:42:16.838 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T05:42:16.838 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:42:16.838 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T05:42:16.839 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T05:42:16.839 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:42:17.353 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:42:17.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T05:42:17.354 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:42:17.354 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:42:17.354 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:42:17.354 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:42:17.381 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:42:17.380+0000 7f9758f1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:42:17.382 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:42:17.382+0000 7f9758f1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:42:17.383 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:42:17.383+0000 7f9758f1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:42:17.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T05:42:17.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:42:17.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:42:17.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:42:17.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:42:17.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:42:17.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:42:17.785 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:42:17.785 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:42:17.785 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:42:18.201 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:42:18.238 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:42:18.238+0000 7f9758f1e640 -1 Falling back to public interface 2024-09-16T05:42:19.202 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:42:19.202 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:42:19.204 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:42:19.204 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:42:19.204 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:42:19.204 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:42:19.619 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:42:20.620 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:42:20.620 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:42:20.621 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:42:20.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:42:20.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:42:20.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:42:20.998 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:42:21.272 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:42:21.272+0000 7f9758f1e640 -1 osd.0 0 log_to_monitors true 2024-09-16T05:42:21.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:42:21.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:42:22.000 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:42:22.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:42:22.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:42:22.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:42:22.477 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:42:23.478 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:42:23.478 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:42:23.479 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:42:23.479 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:42:23.479 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:42:23.479 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:42:23.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:42:24.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:42:24.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:42:24.882 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:42:24.883 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:42:24.883 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:42:24.883 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:42:25.303 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:42:26.304 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:42:26.304 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:42:26.305 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:42:26.305 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:42:26.305 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:42:26.305 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:42:26.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:42:27.720 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:42:27.720 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:42:27.720 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:42:27.721 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:42:27.721 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:42:27.721 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:42:28.132 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:42:29.133 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:42:29.134 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:42:29.134 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:42:29.134 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:42:29.134 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:42:29.134 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:42:29.535 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:42:30.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:42:30.537 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:42:30.537 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:42:30.537 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:42:30.537 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:42:30.537 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:42:30.924 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:42:31.925 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:42:31.925 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:42:31.925 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:42:31.926 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:42:31.926 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:42:31.926 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:42:32.438 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:42:33.439 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:42:33.439 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:42:33.440 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:42:33.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:42:33.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:42:33.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:42:34.119 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:42:35.120 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:42:35.121 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:42:35.121 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:42:35.121 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:42:35.121 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:42:35.122 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:42:35.963 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:42:36.964 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:42:36.964 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:42:36.965 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:42:36.965 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:42:36.965 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:42:36.965 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:42:37.483 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:42:38.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:42:38.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:42:38.486 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:42:38.486 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:42:38.486 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:42:38.486 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:42:39.244 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:42:40.245 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:42:40.245 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:42:40.245 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:42:40.246 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:42:40.246 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:42:40.246 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:42:40.992 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:42:41.994 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:42:41.994 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:42:41.994 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:42:41.995 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:42:41.995 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:42:41.995 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:42:42.812 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:42:43.813 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:42:43.814 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:42:43.814 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:42:43.814 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:42:43.815 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:42:43.815 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:42:44.301 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:42:45.303 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:42:45.303 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:42:45.303 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T05:42:45.304 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T05:42:45.304 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:42:45.304 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:42:45.698 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1762981973,v1:127.0.0.1:6803/1762981973] [v2:127.0.0.1:6804/1762981973,v1:127.0.0.1:6805/1762981973] exists,up 2227ad22-d227-4b1d-b4ba-7fa796ce86e3 2024-09-16T05:42:45.699 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:42:45.699 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:42:45.699 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:42:45.699 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: test_flush_pg_stats: create_rbd_pool 2024-09-16T05:42:45.700 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T05:42:46.097 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' does not exist 2024-09-16T05:42:46.108 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T05:42:46.109 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T05:42:46.571 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' already exists 2024-09-16T05:42:46.582 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T05:42:47.584 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T05:42:48.591 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: test_flush_pg_stats: rados -p rbd put obj /etc/group 2024-09-16T05:42:48.682 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: test_flush_pg_stats: flush_pg_stats 2024-09-16T05:42:48.682 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-16T05:42:48.682 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-16T05:42:49.102 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-09-16T05:42:49.102 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-16T05:42:49.102 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:42:49.102 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T05:42:49.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836487 2024-09-16T05:42:49.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836487 2024-09-16T05:42:49.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487' 2024-09-16T05:42:49.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:42:49.217 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836487 2024-09-16T05:42:49.217 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:42:49.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-16T05:42:49.218 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836487 2024-09-16T05:42:49.218 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:42:49.219 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836487 2024-09-16T05:42:49.219 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-09-16T05:42:49.219 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.0 seq 21474836487 2024-09-16T05:42:49.220 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:42:49.595 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836485 -lt 21474836487 2024-09-16T05:42:49.595 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-16T05:42:50.596 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T05:42:50.596 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:42:51.003 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836485 -lt 21474836487 2024-09-16T05:42:51.004 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-16T05:42:52.004 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-16T05:42:52.005 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:42:52.414 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836488 -lt 21474836487 2024-09-16T05:42:52.414 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: test_flush_pg_stats: local 'jq_filter=.pools | .[] | select(.name == "rbd") | .stats' 2024-09-16T05:42:52.415 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: test_flush_pg_stats: ceph df detail --format=json 2024-09-16T05:42:52.415 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: test_flush_pg_stats: jq '.pools | .[] | select(.name == "rbd") | .stats.stored' 2024-09-16T05:42:52.920 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: test_flush_pg_stats: stored=6809 2024-09-16T05:42:52.921 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: test_flush_pg_stats: ceph df detail --format=json 2024-09-16T05:42:52.921 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: test_flush_pg_stats: jq '.pools | .[] | select(.name == "rbd") | .stats.stored_raw' 2024-09-16T05:42:53.501 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: test_flush_pg_stats: stored_raw=6809 2024-09-16T05:42:53.502 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: test_flush_pg_stats: test 6809 -gt 0 2024-09-16T05:42:53.502 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: test_flush_pg_stats: test 6809 == 6809 2024-09-16T05:42:53.502 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: test_flush_pg_stats: teardown td/ceph-helpers 2024-09-16T05:42:53.502 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:42:53.502 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:42:53.502 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:42:53.502 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:42:53.502 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:42:53.502 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:42:53.502 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:42:53.502 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:42:53.607 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:42:53.607 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:42:53.608 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:42:53.608 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:42:53.610 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:42:53.610 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:42:53.610 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:42:53.610 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:42:53.610 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:42:53.611 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:42:53.611 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:42:53.611 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:42:53.613 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:42:53.613 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:42:53.618 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:42:53.618 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:42:53.618 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:42:53.618 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:42:53.620 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:42:53.620 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:42:53.620 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T05:42:53.620 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_get_config td/ceph-helpers 2024-09-16T05:42:53.620 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: test_get_config: local dir=td/ceph-helpers 2024-09-16T05:42:53.620 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: test_get_config: setup td/ceph-helpers 2024-09-16T05:42:53.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T05:42:53.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T05:42:53.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:42:53.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:42:53.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:42:53.621 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:42:53.621 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:42:53.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:42:53.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:42:53.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:42:53.622 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:42:53.622 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:42:53.622 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:42:53.622 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:42:53.623 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:42:53.623 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:42:53.623 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:42:53.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:42:53.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:42:53.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:42:53.625 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:42:53.625 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:42:53.626 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:42:53.626 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:42:53.627 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:42:53.627 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:42:53.627 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:42:53.627 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:42:53.628 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:42:53.628 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:42:53.628 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T05:42:53.629 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:42:53.630 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:42:53.630 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:42:53.630 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T05:42:53.631 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:42:53.631 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:42:53.631 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:42:53.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T05:42:53.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1140: test_get_config: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:42:53.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T05:42:53.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:42:53.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:42:53.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:42:53.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T05:42:53.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:42:54.019 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:42:54.020 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:42:54.020 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:42:54.020 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:42:54.020 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:42:54.020 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:42:54.020 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:42:54.020 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:42:54.212 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:42:54.214 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:42:54.214 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:42:54.214 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:42:54.214 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:42:54.214 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:42:54.214 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:42:54.214 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:42:54.214 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:42:54.214 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:42:54.214 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:42:54.214 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:42:54.214 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:42:54.215 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:42:54.215 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T05:42:54.292 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:42:54.293 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:42:54.293 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:42:54.293 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:42:54.293 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:42:54.293 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:42:54.293 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:42:54.293 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:42:54.293 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:42:54.293 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:42:54.293 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:42:54.293 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:42:54.293 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:42:54.294 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T05:42:54.372 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: test_get_config: get_config mon a osd_pool_default_size 2024-09-16T05:42:54.372 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:42:54.372 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:42:54.372 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_pool_default_size 2024-09-16T05:42:54.373 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_pool_default_size 2024-09-16T05:42:54.373 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:42:54.373 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:42:54.373 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:42:54.373 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:42:54.373 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:42:54.373 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:42:54.373 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:42:54.373 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:42:54.373 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get osd_pool_default_size 2024-09-16T05:42:54.451 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: test_get_config: test 1 = 1 2024-09-16T05:42:54.451 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: test_get_config: run_mgr td/ceph-helpers x 2024-09-16T05:42:54.451 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T05:42:54.451 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T05:42:54.451 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T05:42:54.451 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T05:42:54.452 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T05:42:54.452 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T05:42:55.299 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T05:42:55.300 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:42:55.300 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:42:55.300 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:42:55.300 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:42:55.300 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:42:55.300 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:42:55.300 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:42:55.301 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:42:55.331 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1143: test_get_config: run_osd td/ceph-helpers 0 --osd_max_scrubs=3 2024-09-16T05:42:55.331 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T05:42:55.331 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:42:55.331 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T05:42:55.331 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:42:55.331 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T05:42:55.332 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T05:42:55.332 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:42:55.332 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:42:55.332 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:42:55.332 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T05:42:55.333 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T05:42:55.333 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:42:55.333 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:42:55.333 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T05:42:55.333 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:42:55.333 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:42:55.333 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:42:55.333 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:42:55.333 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:42:55.333 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:42:55.334 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:42:55.334 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:42:55.334 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:42:55.334 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:42:55.334 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:42:55.334 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T05:42:55.334 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T05:42:55.334 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:42:55.334 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:42:55.335 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:42:55.335 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:42:55.335 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:42:55.335 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_max_scrubs=3 2024-09-16T05:42:55.335 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T05:42:55.335 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:42:55.335 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=25df86c0-9b26-4097-8213-f274d6040c03 2024-09-16T05:42:55.336 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 25df86c0-9b26-4097-8213-f274d6040c03 2024-09-16T05:42:55.336 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 25df86c0-9b26-4097-8213-f274d6040c03' 2024-09-16T05:42:55.336 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:42:55.351 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBfxedmViLeFBAA69/IS1FnhZc0BtY8BT02Qw== 2024-09-16T05:42:55.351 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBfxedmViLeFBAA69/IS1FnhZc0BtY8BT02Qw=="}' 2024-09-16T05:42:55.351 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 25df86c0-9b26-4097-8213-f274d6040c03 -i td/ceph-helpers/0/new.json 2024-09-16T05:42:55.620 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:42:55.630 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T05:42:55.630 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_max_scrubs=3 --mkfs --key AQBfxedmViLeFBAA69/IS1FnhZc0BtY8BT02Qw== --osd-uuid 25df86c0-9b26-4097-8213-f274d6040c03 2024-09-16T05:42:55.660 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:42:55.659+0000 7f06bdd45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:42:55.689 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:42:55.688+0000 7f06bdd45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:42:55.690 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:42:55.689+0000 7f06bdd45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:42:55.690 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:42:55.690+0000 7f06bdd45640 -1 bdev(0x560ccf99fc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-16T05:42:55.690 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:42:55.690+0000 7f06bdd45640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T05:43:00.944 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T05:43:00.945 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:43:00.945 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T05:43:00.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T05:43:00.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:43:01.478 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:43:01.479 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T05:43:01.479 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_max_scrubs=3 2024-09-16T05:43:01.479 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:43:01.479 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:43:01.479 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:43:01.506 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:43:01.505+0000 7f5e78d32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:43:01.532 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:43:01.532+0000 7f5e78d32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:43:01.533 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:43:01.533+0000 7f5e78d32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:43:01.930 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T05:43:01.930 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:43:01.930 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:43:01.930 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:43:01.930 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:43:01.930 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:43:01.930 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:43:01.931 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:43:01.931 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:43:01.931 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:43:02.346 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:43:02.635 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:43:02.634+0000 7f5e78d32640 -1 Falling back to public interface 2024-09-16T05:43:03.347 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:43:03.347 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:43:03.348 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:43:03.348 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:43:03.348 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:43:03.348 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:43:03.764 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:43:04.765 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:43:04.766 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:43:04.766 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:43:04.766 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:43:04.766 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:43:04.767 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:43:05.151 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:43:05.451 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:43:05.450+0000 7f5e78d32640 -1 osd.0 0 log_to_monitors true 2024-09-16T05:43:06.152 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:43:06.153 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:43:06.153 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:43:06.153 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:43:06.153 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:43:06.153 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:43:06.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:43:07.622 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:43:07.622 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:43:07.622 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:43:07.623 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:43:07.623 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:43:07.623 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:43:08.053 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:43:09.054 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:43:09.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:43:09.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:43:09.055 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:43:09.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:43:09.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:43:09.581 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:43:10.583 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:43:10.583 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:43:10.583 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:43:10.584 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:43:10.584 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:43:10.584 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:43:10.995 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:43:11.996 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:43:11.997 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:43:11.997 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:43:11.997 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:43:11.997 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:43:11.998 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:43:12.402 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:43:13.403 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:43:13.403 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:43:13.404 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:43:13.404 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:43:13.404 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:43:13.404 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:43:13.814 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:43:14.815 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:43:14.815 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:43:14.816 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:43:14.816 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:43:14.816 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:43:14.816 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:43:15.272 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:43:16.273 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:43:16.273 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:43:16.273 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:43:16.274 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:43:16.274 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:43:16.274 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:43:16.651 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:43:17.653 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:43:17.653 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:43:17.653 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:43:17.654 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:43:17.654 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:43:17.654 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:43:18.365 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:43:19.366 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:43:19.367 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:43:19.367 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:43:19.367 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:43:19.367 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:43:19.367 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:43:20.162 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:43:21.163 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:43:21.163 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:43:21.163 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:43:21.164 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:43:21.164 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:43:21.164 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:43:21.564 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:43:22.565 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:43:22.565 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:43:22.566 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:43:22.566 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:43:22.566 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:43:22.566 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:43:23.219 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:43:24.220 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:43:24.220 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:43:24.221 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:43:24.221 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:43:24.221 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:43:24.221 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:43:24.929 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:43:25.930 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:43:25.931 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:43:25.931 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:43:25.931 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:43:25.931 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:43:25.931 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:43:27.059 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:43:28.060 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:43:28.060 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:43:28.061 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:43:28.061 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:43:28.061 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:43:28.061 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:43:28.455 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3414918774,v1:127.0.0.1:6803/3414918774] [v2:127.0.0.1:6804/3414918774,v1:127.0.0.1:6805/3414918774] exists,up 25df86c0-9b26-4097-8213-f274d6040c03 2024-09-16T05:43:28.457 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:43:28.457 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:43:28.457 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:43:28.457 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1144: test_get_config: get_config osd 0 osd_max_scrubs 2024-09-16T05:43:28.457 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=osd 2024-09-16T05:43:28.457 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=0 2024-09-16T05:43:28.457 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_max_scrubs 2024-09-16T05:43:28.457 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_max_scrubs 2024-09-16T05:43:28.457 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path osd.0 2024-09-16T05:43:28.458 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:43:28.458 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:43:28.458 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:43:28.458 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:43:28.458 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:43:28.458 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-osd.0.asok 2024-09-16T05:43:28.458 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:43:28.458 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-osd.0.asok config get osd_max_scrubs 2024-09-16T05:43:28.542 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1144: test_get_config: test 3 = 3 2024-09-16T05:43:28.542 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1145: test_get_config: teardown td/ceph-helpers 2024-09-16T05:43:28.542 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:43:28.542 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:43:28.542 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:43:28.543 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:43:28.543 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:43:28.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:43:28.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:43:28.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:43:28.646 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:43:28.646 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:43:28.647 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:43:28.647 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:43:28.648 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:43:28.648 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:43:28.648 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:43:28.650 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:43:28.650 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:43:28.650 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:43:28.651 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:43:28.651 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:43:28.651 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:43:28.651 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:43:28.656 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:43:28.657 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:43:28.657 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:43:28.657 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:43:28.659 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:43:28.659 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:43:28.659 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T05:43:28.659 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_get_is_making_recovery_progress td/ceph-helpers 2024-09-16T05:43:28.659 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: test_get_is_making_recovery_progress: local dir=td/ceph-helpers 2024-09-16T05:43:28.659 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: test_get_is_making_recovery_progress: setup td/ceph-helpers 2024-09-16T05:43:28.659 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T05:43:28.659 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T05:43:28.659 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:43:28.659 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:43:28.659 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:43:28.660 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:43:28.660 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:43:28.660 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:43:28.660 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:43:28.660 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:43:28.660 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:43:28.661 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:43:28.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:43:28.661 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:43:28.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:43:28.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:43:28.662 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:43:28.663 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:43:28.663 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:43:28.663 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:43:28.663 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:43:28.664 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:43:28.665 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:43:28.665 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:43:28.666 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:43:28.666 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:43:28.666 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:43:28.666 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:43:28.667 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:43:28.667 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:43:28.667 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T05:43:28.668 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:43:28.668 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:43:28.668 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:43:28.668 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T05:43:28.671 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:43:28.671 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:43:28.671 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:43:28.671 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T05:43:28.671 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1347: test_get_is_making_recovery_progress: run_mon td/ceph-helpers a 2024-09-16T05:43:28.671 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T05:43:28.671 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:43:28.671 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:43:28.671 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:43:28.671 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T05:43:28.671 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-16T05:43:29.050 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:43:29.050 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:43:29.050 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:43:29.050 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:43:29.050 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:43:29.050 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:43:29.050 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:43:29.050 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T05:43:29.242 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:43:29.243 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:43:29.243 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:43:29.243 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:43:29.243 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:43:29.244 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:43:29.244 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:43:29.244 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:43:29.244 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:43:29.244 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:43:29.244 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:43:29.244 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:43:29.244 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:43:29.244 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:43:29.244 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T05:43:29.323 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:43:29.323 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:43:29.323 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:43:29.323 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:43:29.323 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:43:29.324 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:43:29.324 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:43:29.324 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:43:29.324 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:43:29.324 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:43:29.324 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:43:29.324 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:43:29.324 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:43:29.324 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T05:43:29.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1348: test_get_is_making_recovery_progress: run_mgr td/ceph-helpers x 2024-09-16T05:43:29.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T05:43:29.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T05:43:29.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T05:43:29.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T05:43:29.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T05:43:29.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T05:43:30.238 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T05:43:30.238 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:43:30.238 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:43:30.238 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:43:30.238 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:43:30.238 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:43:30.238 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:43:30.239 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:43:30.239 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:43:30.266 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1349: test_get_is_making_recovery_progress: get_is_making_recovery_progress 2024-09-16T05:43:30.266 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T05:43:30.266 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T05:43:30.266 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T05:43:30.266 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T05:43:30.266 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T05:43:30.267 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T05:43:30.501 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T05:43:30.501 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T05:43:30.501 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1350: test_get_is_making_recovery_progress: teardown td/ceph-helpers 2024-09-16T05:43:30.501 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:43:30.501 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:43:30.501 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:43:30.501 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:43:30.501 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:43:30.502 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:43:30.502 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:43:30.502 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:43:30.606 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:43:30.606 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:43:30.607 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:43:30.607 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:43:30.608 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:43:30.608 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:43:30.608 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:43:30.609 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:43:30.609 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:43:30.609 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:43:30.609 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:43:30.610 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:43:30.611 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:43:30.611 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:43:30.613 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:43:30.613 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:43:30.613 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:43:30.613 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:43:30.615 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:43:30.615 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:43:30.615 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T05:43:30.615 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_get_last_scrub_stamp td/ceph-helpers 2024-09-16T05:43:30.615 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1533: test_get_last_scrub_stamp: local dir=td/ceph-helpers 2024-09-16T05:43:30.615 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1535: test_get_last_scrub_stamp: setup td/ceph-helpers 2024-09-16T05:43:30.615 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T05:43:30.615 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T05:43:30.615 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:43:30.615 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:43:30.615 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:43:30.615 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:43:30.616 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:43:30.616 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:43:30.616 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:43:30.616 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:43:30.616 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:43:30.616 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:43:30.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:43:30.617 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:43:30.618 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:43:30.618 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:43:30.618 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:43:30.619 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:43:30.619 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:43:30.619 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:43:30.619 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:43:30.620 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:43:30.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:43:30.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:43:30.622 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:43:30.622 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:43:30.622 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:43:30.622 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:43:30.623 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:43:30.623 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:43:30.623 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T05:43:30.624 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:43:30.624 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:43:30.624 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:43:30.625 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T05:43:30.626 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:43:30.626 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:43:30.627 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:43:30.627 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T05:43:30.627 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1536: test_get_last_scrub_stamp: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:43:30.627 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T05:43:30.627 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:43:30.627 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:43:30.627 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:43:30.627 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T05:43:30.627 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:43:31.073 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:43:31.073 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:43:31.073 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:43:31.073 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:43:31.073 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:43:31.073 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:43:31.073 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:43:31.073 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:43:31.265 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:43:31.267 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:43:31.267 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:43:31.267 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:43:31.267 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:43:31.267 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:43:31.267 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:43:31.267 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:43:31.267 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:43:31.267 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:43:31.267 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:43:31.268 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:43:31.268 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:43:31.268 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:43:31.268 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T05:43:31.346 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:43:31.346 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:43:31.346 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:43:31.346 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:43:31.346 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:43:31.346 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:43:31.346 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:43:31.346 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:43:31.347 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:43:31.347 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:43:31.347 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:43:31.347 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:43:31.347 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:43:31.347 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T05:43:31.424 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1537: test_get_last_scrub_stamp: run_mgr td/ceph-helpers x 2024-09-16T05:43:31.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T05:43:31.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T05:43:31.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T05:43:31.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T05:43:31.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T05:43:31.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T05:43:32.270 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T05:43:32.270 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:43:32.270 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:43:32.270 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:43:32.270 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:43:32.270 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:43:32.270 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:43:32.270 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:43:32.271 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:43:32.303 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1538: test_get_last_scrub_stamp: run_osd td/ceph-helpers 0 2024-09-16T05:43:32.303 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T05:43:32.303 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:43:32.303 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T05:43:32.303 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:43:32.303 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T05:43:32.303 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T05:43:32.304 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:43:32.304 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:43:32.304 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:43:32.304 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T05:43:32.304 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T05:43:32.304 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:43:32.304 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:43:32.304 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T05:43:32.304 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:43:32.304 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:43:32.304 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:43:32.304 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:43:32.305 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:43:32.305 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:43:32.305 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:43:32.305 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:43:32.305 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:43:32.305 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:43:32.305 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:43:32.305 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T05:43:32.305 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T05:43:32.305 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:43:32.305 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:43:32.305 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:43:32.305 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:43:32.305 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:43:32.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:43:32.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T05:43:32.306 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:43:32.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ef4f3981-ba15-4c72-be9f-1e5d5b95c584 2024-09-16T05:43:32.306 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 ef4f3981-ba15-4c72-be9f-1e5d5b95c584 2024-09-16T05:43:32.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ef4f3981-ba15-4c72-be9f-1e5d5b95c584' 2024-09-16T05:43:32.307 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:43:32.322 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCExedm9dMsExAAOsDqBUvabbesD4TQ4/RGlg== 2024-09-16T05:43:32.323 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCExedm9dMsExAAOsDqBUvabbesD4TQ4/RGlg=="}' 2024-09-16T05:43:32.323 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ef4f3981-ba15-4c72-be9f-1e5d5b95c584 -i td/ceph-helpers/0/new.json 2024-09-16T05:43:32.590 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:43:32.600 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T05:43:32.600 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCExedm9dMsExAAOsDqBUvabbesD4TQ4/RGlg== --osd-uuid ef4f3981-ba15-4c72-be9f-1e5d5b95c584 2024-09-16T05:43:32.630 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:43:32.629+0000 7f57d5ce4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:43:32.651 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:43:32.650+0000 7f57d5ce4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:43:32.652 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:43:32.652+0000 7f57d5ce4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:43:32.653 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:43:32.652+0000 7f57d5ce4640 -1 bdev(0x560bcba79c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-16T05:43:32.653 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:43:32.652+0000 7f57d5ce4640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T05:43:37.987 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T05:43:37.987 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:43:37.988 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T05:43:37.988 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T05:43:37.988 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:43:38.548 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T05:43:38.549 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:43:38.549 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:43:38.550 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:43:38.550 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:43:38.550 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:43:38.575 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:43:38.574+0000 7f5caa424640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:43:38.577 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:43:38.576+0000 7f5caa424640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:43:38.578 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:43:38.577+0000 7f5caa424640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:43:38.922 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T05:43:38.922 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:43:38.922 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:43:38.922 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:43:38.922 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:43:38.923 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:43:38.923 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:43:38.923 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:43:38.923 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:43:38.923 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:43:39.297 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:43:40.298 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:43:40.298 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:43:40.298 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:43:40.299 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:43:40.299 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:43:40.299 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:43:40.443 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:43:40.443+0000 7f5caa424640 -1 Falling back to public interface 2024-09-16T05:43:40.684 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:43:41.683 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:43:41.683 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:43:41.684 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:43:41.684 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:43:41.684 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:43:41.684 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:43:42.104 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:43:43.064 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:43:43.063+0000 7f5caa424640 -1 osd.0 0 log_to_monitors true 2024-09-16T05:43:43.104 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:43:43.104 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:43:43.105 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:43:43.105 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:43:43.105 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:43:43.105 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:43:43.675 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:43:44.676 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:43:44.677 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:43:44.677 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:43:44.677 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:43:44.677 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:43:44.677 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:43:45.109 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:43:46.109 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:43:46.110 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:43:46.110 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:43:46.110 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:43:46.111 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:43:46.111 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:43:46.526 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:43:47.527 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:43:47.527 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:43:47.528 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:43:47.528 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:43:47.528 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:43:47.528 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:43:47.975 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:43:48.977 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:43:48.977 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:43:48.977 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:43:48.978 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:43:48.978 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:43:48.978 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:43:49.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:43:50.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:43:50.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:43:50.401 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:43:50.402 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:43:50.402 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:43:50.402 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:43:50.831 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:43:51.832 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:43:51.832 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:43:51.833 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:43:51.833 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:43:51.834 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:43:51.834 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:43:52.245 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:43:53.246 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:43:53.247 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:43:53.247 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:43:53.247 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:43:53.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:43:53.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:43:53.627 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:43:54.628 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:43:54.629 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:43:54.629 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:43:54.629 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:43:54.630 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:43:54.630 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:43:55.433 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:43:56.434 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:43:56.434 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:43:56.434 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:43:56.435 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:43:56.435 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:43:56.435 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:43:57.526 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:43:58.527 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:43:58.527 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:43:58.528 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:43:58.528 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:43:58.528 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:43:58.528 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:43:59.308 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:44:00.308 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:44:00.308 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:44:00.308 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:44:00.309 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:44:00.309 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:44:00.309 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:44:01.140 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:44:02.141 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:44:02.141 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:44:02.142 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:44:02.142 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:44:02.143 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:44:02.143 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:44:02.904 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:44:03.906 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:44:03.906 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:44:03.907 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:44:03.907 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:44:03.908 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:44:03.908 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:44:04.595 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:44:05.596 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:44:05.596 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:44:05.597 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:44:05.597 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:44:05.597 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:44:05.597 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:44:06.025 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/880709067,v1:127.0.0.1:6803/880709067] [v2:127.0.0.1:6804/880709067,v1:127.0.0.1:6805/880709067] exists,up ef4f3981-ba15-4c72-be9f-1e5d5b95c584 2024-09-16T05:44:06.025 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:44:06.025 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:44:06.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:44:06.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1539: test_get_last_scrub_stamp: create_rbd_pool 2024-09-16T05:44:06.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T05:44:06.401 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' does not exist 2024-09-16T05:44:06.413 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T05:44:06.413 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T05:44:06.881 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' already exists 2024-09-16T05:44:06.889 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T05:44:07.890 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T05:44:09.014 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1540: test_get_last_scrub_stamp: wait_for_clean 2024-09-16T05:44:09.014 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T05:44:09.014 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T05:44:09.014 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T05:44:09.014 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T05:44:09.015 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T05:44:09.015 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T05:44:09.015 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T05:44:09.015 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T05:44:09.015 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T05:44:09.109 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T05:44:09.109 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T05:44:09.110 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T05:44:09.110 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T05:44:09.110 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-16T05:44:09.110 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-16T05:44:09.507 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-09-16T05:44:09.507 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-16T05:44:09.507 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:44:09.507 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T05:44:09.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836486 2024-09-16T05:44:09.622 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836486 2024-09-16T05:44:09.622 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836486' 2024-09-16T05:44:09.622 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:44:09.622 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836486 2024-09-16T05:44:09.622 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:44:09.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-16T05:44:09.624 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836486 2024-09-16T05:44:09.624 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:44:09.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836486 2024-09-16T05:44:09.624 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.0 seq 21474836486 2024-09-16T05:44:09.625 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-09-16T05:44:09.625 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:44:10.033 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836484 -lt 21474836486 2024-09-16T05:44:10.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-16T05:44:11.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T05:44:11.035 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:44:11.497 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836486 -lt 21474836486 2024-09-16T05:44:11.498 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T05:44:11.498 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:44:11.498 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:44:12.089 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T05:44:12.090 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T05:44:12.090 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T05:44:12.090 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T05:44:12.090 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T05:44:12.090 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T05:44:12.090 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T05:44:12.090 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T05:44:12.511 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T05:44:12.512 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T05:44:12.512 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:44:12.512 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:44:13.013 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T05:44:13.013 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T05:44:13.014 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T05:44:13.014 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1541: test_get_last_scrub_stamp: get_last_scrub_stamp 1.0 2024-09-16T05:44:13.014 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T05:44:13.014 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T05:44:13.014 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T05:44:13.014 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T05:44:13.413 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1541: test_get_last_scrub_stamp: stamp=2024-09-16T05:44:06.727682+0000 2024-09-16T05:44:13.413 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1542: test_get_last_scrub_stamp: test -n 2024-09-16T05:44:06.727682+0000 2024-09-16T05:44:13.413 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1543: test_get_last_scrub_stamp: teardown td/ceph-helpers 2024-09-16T05:44:13.413 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:44:13.414 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:44:13.414 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:44:13.414 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:44:13.414 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:44:13.414 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:44:13.414 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:44:13.414 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:44:13.518 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:44:13.519 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:44:13.519 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:44:13.520 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:44:13.521 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:44:13.521 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:44:13.521 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:44:13.522 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:44:13.522 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:44:13.522 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:44:13.522 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:44:13.523 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:44:13.524 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:44:13.524 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:44:13.529 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:44:13.530 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:44:13.530 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:44:13.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:44:13.531 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:44:13.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:44:13.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T05:44:13.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_get_not_primary td/ceph-helpers 2024-09-16T05:44:13.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: test_get_not_primary: local dir=td/ceph-helpers 2024-09-16T05:44:13.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: test_get_not_primary: setup td/ceph-helpers 2024-09-16T05:44:13.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T05:44:13.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T05:44:13.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:44:13.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:44:13.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:44:13.532 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:44:13.532 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:44:13.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:44:13.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:44:13.533 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:44:13.533 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:44:13.533 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:44:13.534 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:44:13.534 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:44:13.535 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:44:13.535 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:44:13.535 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:44:13.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:44:13.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:44:13.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:44:13.536 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:44:13.537 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:44:13.538 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:44:13.538 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:44:13.539 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:44:13.539 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:44:13.539 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:44:13.539 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:44:13.540 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:44:13.540 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:44:13.540 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T05:44:13.541 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:44:13.541 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:44:13.541 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:44:13.541 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T05:44:13.543 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:44:13.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:44:13.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:44:13.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T05:44:13.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: test_get_not_primary: run_mon td/ceph-helpers a --osd_pool_default_size=2 2024-09-16T05:44:13.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T05:44:13.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:44:13.544 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:44:13.544 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:44:13.544 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T05:44:13.544 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2024-09-16T05:44:13.920 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:44:13.920 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:44:13.920 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:44:13.920 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:44:13.921 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:44:13.921 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:44:13.921 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:44:13.921 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-16T05:44:14.096 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:44:14.098 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:44:14.098 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:44:14.098 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:44:14.098 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:44:14.098 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:44:14.098 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:44:14.098 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:44:14.098 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:44:14.098 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:44:14.098 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:44:14.099 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:44:14.099 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:44:14.099 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:44:14.099 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T05:44:14.178 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:44:14.178 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:44:14.178 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:44:14.178 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:44:14.178 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:44:14.178 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:44:14.178 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:44:14.179 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:44:14.179 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:44:14.179 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:44:14.179 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:44:14.179 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:44:14.179 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:44:14.179 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T05:44:14.256 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: test_get_not_primary: run_mgr td/ceph-helpers x 2024-09-16T05:44:14.256 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T05:44:14.256 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T05:44:14.256 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T05:44:14.257 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T05:44:14.257 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T05:44:14.257 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T05:44:15.101 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T05:44:15.101 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:44:15.101 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:44:15.102 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:44:15.102 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:44:15.102 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:44:15.102 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:44:15.102 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:44:15.103 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:44:15.132 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1243: test_get_not_primary: run_osd td/ceph-helpers 0 2024-09-16T05:44:15.132 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T05:44:15.132 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:44:15.132 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T05:44:15.132 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:44:15.132 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T05:44:15.133 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T05:44:15.133 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:44:15.133 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:44:15.133 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:44:15.133 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T05:44:15.133 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T05:44:15.133 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:44:15.133 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:44:15.133 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T05:44:15.133 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:44:15.133 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:44:15.133 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:44:15.134 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:44:15.134 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:44:15.134 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:44:15.134 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:44:15.134 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:44:15.134 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:44:15.134 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:44:15.134 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:44:15.134 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T05:44:15.134 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T05:44:15.134 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:44:15.134 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:44:15.135 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:44:15.135 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:44:15.135 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:44:15.135 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:44:15.135 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T05:44:15.135 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:44:15.136 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e209fbf5-087a-4039-80f1-4d4ecbc15085 2024-09-16T05:44:15.136 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e209fbf5-087a-4039-80f1-4d4ecbc15085' 2024-09-16T05:44:15.136 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 e209fbf5-087a-4039-80f1-4d4ecbc15085 2024-09-16T05:44:15.136 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:44:15.152 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCvxedmF9UBCRAAPVqQp4+8wlT6I7g6+Wnqhg== 2024-09-16T05:44:15.152 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCvxedmF9UBCRAAPVqQp4+8wlT6I7g6+Wnqhg=="}' 2024-09-16T05:44:15.152 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e209fbf5-087a-4039-80f1-4d4ecbc15085 -i td/ceph-helpers/0/new.json 2024-09-16T05:44:15.446 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:44:15.457 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T05:44:15.457 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCvxedmF9UBCRAAPVqQp4+8wlT6I7g6+Wnqhg== --osd-uuid e209fbf5-087a-4039-80f1-4d4ecbc15085 2024-09-16T05:44:15.486 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:44:15.484+0000 7f63f2239640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:44:15.515 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:44:15.513+0000 7f63f2239640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:44:15.516 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:44:15.514+0000 7f63f2239640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:44:15.516 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:44:15.515+0000 7f63f2239640 -1 bdev(0x561adb079c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-16T05:44:15.517 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:44:15.515+0000 7f63f2239640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T05:44:20.937 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T05:44:20.937 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:44:20.938 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T05:44:20.938 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T05:44:20.939 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:44:21.454 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:44:21.455 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T05:44:21.455 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:44:21.455 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:44:21.455 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:44:21.455 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:44:21.481 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:44:21.481+0000 7f1c2ae0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:44:21.503 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:44:21.502+0000 7f1c2ae0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:44:21.504 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:44:21.503+0000 7f1c2ae0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:44:21.908 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T05:44:21.908 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:44:21.908 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:44:21.908 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:44:21.908 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:44:21.909 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:44:21.909 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:44:21.909 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:44:21.909 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:44:21.909 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:44:22.279 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:44:23.097 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:44:23.097+0000 7f1c2ae0b640 -1 Falling back to public interface 2024-09-16T05:44:23.280 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:44:23.281 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:44:23.281 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:44:23.281 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:44:23.281 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:44:23.281 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:44:23.655 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:44:24.656 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:44:24.656 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:44:24.657 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:44:24.657 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:44:24.657 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:44:24.658 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:44:25.069 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:44:25.773 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:44:25.772+0000 7f1c2ae0b640 -1 osd.0 0 log_to_monitors true 2024-09-16T05:44:26.070 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:44:26.071 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:44:26.071 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:44:26.071 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:44:26.071 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:44:26.071 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:44:26.549 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:44:27.550 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:44:27.550 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:44:27.550 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:44:27.551 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:44:27.551 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:44:27.551 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:44:27.953 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:44:28.954 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:44:28.954 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:44:28.954 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:44:28.955 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:44:28.955 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:44:28.955 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:44:29.332 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:44:30.333 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:44:30.333 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:44:30.334 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:44:30.334 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:44:30.334 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:44:30.334 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:44:30.800 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:44:31.801 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:44:31.802 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:44:31.802 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:44:31.802 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:44:31.803 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:44:31.803 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:44:32.183 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:44:33.184 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:44:33.184 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:44:33.185 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:44:33.185 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:44:33.185 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:44:33.185 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:44:33.648 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:44:34.650 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:44:34.650 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:44:34.650 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:44:34.651 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:44:34.651 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:44:34.651 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:44:35.063 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:44:36.065 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:44:36.065 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:44:36.065 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:44:36.066 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:44:36.066 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:44:36.066 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:44:36.573 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:44:37.575 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:44:37.575 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:44:37.575 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:44:37.576 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:44:37.576 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:44:37.576 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:44:38.241 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:44:39.238 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:44:39.239 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:44:39.239 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:44:39.239 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:44:39.239 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:44:39.240 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:44:40.281 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:44:41.282 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:44:41.282 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:44:41.282 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:44:41.283 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:44:41.283 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:44:41.283 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:44:42.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:44:43.022 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:44:43.022 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:44:43.023 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:44:43.023 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:44:43.023 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:44:43.023 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:44:43.467 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:44:44.469 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:44:44.469 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:44:44.469 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:44:44.470 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:44:44.470 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:44:44.470 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:44:45.157 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:44:46.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:44:46.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:44:46.160 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:44:46.160 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:44:46.161 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:44:46.161 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:44:46.980 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:44:47.981 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:44:47.981 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:44:47.982 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:44:47.982 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:44:47.982 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:44:47.982 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:44:48.464 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:44:49.465 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:44:49.465 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:44:49.466 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T05:44:49.466 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T05:44:49.467 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:44:49.467 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:44:49.885 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4173344754,v1:127.0.0.1:6803/4173344754] [v2:127.0.0.1:6804/4173344754,v1:127.0.0.1:6805/4173344754] exists,up e209fbf5-087a-4039-80f1-4d4ecbc15085 2024-09-16T05:44:49.893 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:44:49.893 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:44:49.893 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:44:49.893 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: test_get_not_primary: run_osd td/ceph-helpers 1 2024-09-16T05:44:49.894 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T05:44:49.894 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:44:49.894 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T05:44:49.894 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:44:49.894 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-09-16T05:44:49.894 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T05:44:49.894 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:44:49.894 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:44:49.894 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:44:49.894 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-16T05:44:49.894 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-16T05:44:49.895 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:44:49.895 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:44:49.895 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T05:44:49.895 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:44:49.895 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:44:49.895 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:44:49.895 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:44:49.895 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:44:49.895 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:44:49.895 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:44:49.895 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:44:49.895 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:44:49.895 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:44:49.895 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:44:49.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T05:44:49.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T05:44:49.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:44:49.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:44:49.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:44:49.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:44:49.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:44:49.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:44:49.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-09-16T05:44:49.896 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:44:49.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=80efc7cf-fa9c-4630-8811-c765180d47d7 2024-09-16T05:44:49.897 INFO:tasks.workunit.client.0.smithi178.stdout:add osd1 80efc7cf-fa9c-4630-8811-c765180d47d7 2024-09-16T05:44:49.898 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 80efc7cf-fa9c-4630-8811-c765180d47d7' 2024-09-16T05:44:49.898 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:44:49.909 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDRxedmz+wlNhAAXuPEhUbBxYTjPj+0JRB5aw== 2024-09-16T05:44:49.909 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDRxedmz+wlNhAAXuPEhUbBxYTjPj+0JRB5aw=="}' 2024-09-16T05:44:49.909 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 80efc7cf-fa9c-4630-8811-c765180d47d7 -i td/ceph-helpers/1/new.json 2024-09-16T05:44:50.379 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:44:50.391 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-09-16T05:44:50.392 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDRxedmz+wlNhAAXuPEhUbBxYTjPj+0JRB5aw== --osd-uuid 80efc7cf-fa9c-4630-8811-c765180d47d7 2024-09-16T05:44:50.420 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:44:50.420+0000 7efe2e62c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:44:50.435 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:44:50.434+0000 7efe2e62c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:44:50.436 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:44:50.436+0000 7efe2e62c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:44:50.436 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:44:50.436+0000 7efe2e62c640 -1 bdev(0x55bb79a59c00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2024-09-16T05:44:50.436 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:44:50.436+0000 7efe2e62c640 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-16T05:44:56.465 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-16T05:44:56.466 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:44:56.466 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd1 key to auth repository 2024-09-16T05:44:56.467 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T05:44:56.467 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:44:56.983 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.1 2024-09-16T05:44:56.983 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T05:44:56.983 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:44:56.984 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:44:56.984 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:44:56.984 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:44:57.010 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:44:57.009+0000 7f29d6529640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:44:57.011 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:44:57.011+0000 7f29d6529640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:44:57.013 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:44:57.012+0000 7f29d6529640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:44:57.372 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T05:44:57.372 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:44:57.372 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T05:44:57.372 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:44:57.372 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:44:57.372 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:44:57.372 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:44:57.373 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:44:57.373 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:44:57.373 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:44:57.813 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:44:58.103 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:44:58.103+0000 7f29d6529640 -1 Falling back to public interface 2024-09-16T05:44:58.815 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:44:58.815 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:44:58.815 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:44:58.816 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:44:58.816 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:44:58.816 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:44:59.231 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:45:00.232 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:45:00.232 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:45:00.233 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:45:00.233 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:45:00.233 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:45:00.233 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:45:00.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:45:00.918 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:45:00.917+0000 7f29d6529640 -1 osd.1 0 log_to_monitors true 2024-09-16T05:45:01.622 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:45:01.622 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:45:01.623 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:45:01.623 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:45:01.623 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:45:01.623 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:45:02.097 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:45:03.098 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:45:03.099 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:45:03.099 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:45:03.100 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:45:03.100 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:45:03.100 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:45:03.616 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:45:04.616 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:45:04.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:45:04.617 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:45:04.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:45:04.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:45:04.618 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:45:05.117 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:45:06.119 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:45:06.119 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:45:06.119 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:45:06.119 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:45:06.120 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:45:06.120 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:45:06.527 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:45:07.528 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:45:07.528 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:45:07.529 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:45:07.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:45:07.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:45:07.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:45:08.091 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:45:09.093 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:45:09.093 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:45:09.093 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:45:09.094 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:45:09.095 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:45:09.095 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:45:09.590 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:45:10.591 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:45:10.591 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:45:10.592 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:45:10.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:45:10.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:45:10.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:45:11.128 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:45:12.129 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:45:12.130 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:45:12.130 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:45:12.130 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:45:12.130 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:45:12.131 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:45:12.551 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:45:13.552 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:45:13.552 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:45:13.553 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:45:13.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:45:13.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:45:13.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:45:14.119 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:45:15.120 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:45:15.121 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:45:15.121 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:45:15.121 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:45:15.121 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:45:15.122 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:45:15.616 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:45:16.618 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:45:16.618 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:45:16.618 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:45:16.618 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:45:16.619 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:45:16.619 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:45:17.291 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:45:18.292 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:45:18.292 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:45:18.292 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:45:18.293 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:45:18.293 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:45:18.293 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:45:19.069 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:45:20.071 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:45:20.071 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:45:20.071 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:45:20.072 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:45:20.072 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:45:20.072 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:45:20.733 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:45:21.735 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:45:21.735 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:45:21.735 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:45:21.736 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:45:21.736 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:45:21.736 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:45:22.476 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:45:23.477 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:45:23.478 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:45:23.478 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:45:23.478 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:45:23.478 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:45:23.478 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:45:24.343 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:45:25.344 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:45:25.345 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:45:25.345 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T05:45:25.345 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T05:45:25.345 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:45:25.346 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:45:25.765 INFO:tasks.workunit.client.0.smithi178.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3993366918,v1:127.0.0.1:6811/3993366918] [v2:127.0.0.1:6812/3993366918,v1:127.0.0.1:6813/3993366918] exists,up 80efc7cf-fa9c-4630-8811-c765180d47d7 2024-09-16T05:45:25.765 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:45:25.765 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:45:25.766 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:45:25.766 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: test_get_not_primary: create_rbd_pool 2024-09-16T05:45:25.766 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T05:45:26.128 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' does not exist 2024-09-16T05:45:26.138 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T05:45:26.138 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T05:45:26.606 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' already exists 2024-09-16T05:45:26.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T05:45:27.619 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T05:45:29.450 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1246: test_get_not_primary: wait_for_clean 2024-09-16T05:45:29.450 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T05:45:29.450 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T05:45:29.450 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T05:45:29.450 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T05:45:29.451 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T05:45:29.451 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T05:45:29.451 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T05:45:29.451 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T05:45:29.451 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T05:45:29.545 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T05:45:29.546 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T05:45:29.546 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T05:45:29.546 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T05:45:29.546 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-16T05:45:29.546 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-16T05:45:29.968 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-16T05:45:29.968 INFO:tasks.workunit.client.0.smithi178.stderr:1' 2024-09-16T05:45:29.968 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-16T05:45:29.968 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:45:29.969 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T05:45:30.110 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836495 2024-09-16T05:45:30.110 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836495 2024-09-16T05:45:30.110 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495' 2024-09-16T05:45:30.110 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:45:30.110 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T05:45:30.223 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672965 2024-09-16T05:45:30.223 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672965 2024-09-16T05:45:30.223 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495 1-42949672965' 2024-09-16T05:45:30.223 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:45:30.223 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836495 2024-09-16T05:45:30.223 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:45:30.224 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-16T05:45:30.224 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836495 2024-09-16T05:45:30.224 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:45:30.225 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836495 2024-09-16T05:45:30.225 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2024-09-16T05:45:30.226 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.0 seq 21474836495 2024-09-16T05:45:30.226 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:45:30.636 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836495 2024-09-16T05:45:30.637 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-16T05:45:31.637 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T05:45:31.638 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:45:32.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836495 -lt 21474836495 2024-09-16T05:45:32.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:45:32.056 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672965 2024-09-16T05:45:32.056 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:45:32.056 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-09-16T05:45:32.057 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672965 2024-09-16T05:45:32.057 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:45:32.058 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672965 2024-09-16T05:45:32.058 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-09-16T05:45:32.058 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.1 seq 42949672965 2024-09-16T05:45:32.059 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T05:45:32.472 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672966 -lt 42949672965 2024-09-16T05:45:32.473 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T05:45:32.473 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:45:32.473 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:45:32.973 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T05:45:32.974 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T05:45:32.974 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T05:45:32.974 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T05:45:32.974 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T05:45:32.974 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T05:45:32.974 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T05:45:32.974 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T05:45:33.363 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T05:45:33.364 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T05:45:33.364 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:45:33.364 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:45:33.862 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T05:45:33.863 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T05:45:33.863 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T05:45:33.863 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1247: test_get_not_primary: get_primary rbd GROUP 2024-09-16T05:45:33.863 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-09-16T05:45:33.863 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=GROUP 2024-09-16T05:45:33.863 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd GROUP 2024-09-16T05:45:33.863 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-16T05:45:34.295 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1247: test_get_not_primary: local primary=1 2024-09-16T05:45:34.295 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: test_get_not_primary: get_not_primary rbd GROUP 2024-09-16T05:45:34.295 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-09-16T05:45:34.295 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=GROUP 2024-09-16T05:45:34.295 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd GROUP 2024-09-16T05:45:34.296 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-09-16T05:45:34.296 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=GROUP 2024-09-16T05:45:34.296 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd GROUP 2024-09-16T05:45:34.296 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-16T05:45:34.727 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-16T05:45:34.728 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd GROUP 2024-09-16T05:45:34.728 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-16T05:45:35.121 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: test_get_not_primary: local not_primary=0 2024-09-16T05:45:35.121 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: test_get_not_primary: test 0 '!=' 1 2024-09-16T05:45:35.121 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: test_get_not_primary: test 0 = 0 -o 0 = 1 2024-09-16T05:45:35.121 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: test_get_not_primary: teardown td/ceph-helpers 2024-09-16T05:45:35.121 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:45:35.121 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:45:35.121 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:45:35.121 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:45:35.122 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:45:35.122 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:45:35.122 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:45:35.122 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:45:35.227 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:45:35.227 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:45:35.227 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:45:35.228 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:45:35.229 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:45:35.229 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:45:35.229 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:45:35.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:45:35.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:45:35.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:45:35.231 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:45:35.231 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:45:35.232 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:45:35.232 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:45:35.242 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:45:35.242 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:45:35.242 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:45:35.242 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:45:35.244 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:45:35.244 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:45:35.244 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T05:45:35.244 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_get_num_active_clean td/ceph-helpers 2024-09-16T05:45:35.244 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1372: test_get_num_active_clean: local dir=td/ceph-helpers 2024-09-16T05:45:35.244 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1374: test_get_num_active_clean: setup td/ceph-helpers 2024-09-16T05:45:35.244 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T05:45:35.244 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T05:45:35.244 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:45:35.244 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:45:35.245 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:45:35.245 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:45:35.245 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:45:35.245 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:45:35.245 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:45:35.245 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:45:35.245 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:45:35.245 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:45:35.246 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:45:35.246 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:45:35.247 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:45:35.247 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:45:35.247 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:45:35.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:45:35.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:45:35.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:45:35.249 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:45:35.249 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:45:35.250 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:45:35.250 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:45:35.251 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:45:35.251 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:45:35.251 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:45:35.251 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:45:35.252 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:45:35.252 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:45:35.252 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T05:45:35.253 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:45:35.253 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:45:35.253 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:45:35.254 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T05:45:35.255 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:45:35.255 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:45:35.255 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:45:35.256 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T05:45:35.256 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1375: test_get_num_active_clean: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:45:35.256 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T05:45:35.256 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:45:35.256 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:45:35.256 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:45:35.256 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T05:45:35.256 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:45:35.659 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:45:35.659 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:45:35.659 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:45:35.660 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:45:35.660 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:45:35.660 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:45:35.660 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:45:35.660 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:45:35.851 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:45:35.853 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:45:35.853 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:45:35.853 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:45:35.853 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:45:35.853 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:45:35.853 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:45:35.853 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:45:35.854 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:45:35.854 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:45:35.854 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:45:35.854 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:45:35.854 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:45:35.854 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:45:35.854 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T05:45:35.932 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:45:35.933 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:45:35.933 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:45:35.933 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:45:35.933 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:45:35.933 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:45:35.933 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:45:35.933 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:45:35.933 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:45:35.933 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:45:35.933 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:45:35.933 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:45:35.933 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:45:35.934 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T05:45:36.010 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1376: test_get_num_active_clean: run_mgr td/ceph-helpers x 2024-09-16T05:45:36.011 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T05:45:36.011 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T05:45:36.011 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T05:45:36.011 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T05:45:36.011 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T05:45:36.011 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T05:45:36.856 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T05:45:36.856 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:45:36.856 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:45:36.857 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:45:36.857 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:45:36.857 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:45:36.857 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:45:36.857 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:45:36.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:45:36.887 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1377: test_get_num_active_clean: run_osd td/ceph-helpers 0 2024-09-16T05:45:36.887 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T05:45:36.887 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:45:36.887 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T05:45:36.887 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:45:36.887 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T05:45:36.887 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T05:45:36.888 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:45:36.888 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:45:36.888 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:45:36.888 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T05:45:36.888 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T05:45:36.888 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:45:36.888 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:45:36.888 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T05:45:36.888 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:45:36.888 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:45:36.888 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:45:36.888 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:45:36.888 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:45:36.889 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:45:36.890 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:45:36.890 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:45:36.890 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:45:36.890 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:45:36.890 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:45:36.890 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T05:45:36.890 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T05:45:36.890 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:45:36.890 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:45:36.890 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:45:36.891 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:45:36.891 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:45:36.891 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:45:36.891 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T05:45:36.891 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:45:36.892 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c6c76286-95d0-4a88-a09c-e0d1bfb2bae4 2024-09-16T05:45:36.892 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c6c76286-95d0-4a88-a09c-e0d1bfb2bae4' 2024-09-16T05:45:36.892 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 c6c76286-95d0-4a88-a09c-e0d1bfb2bae4 2024-09-16T05:45:36.892 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:45:36.907 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAAxudmhFEMNhAADcZ5FsWhkYDY1+jeW3527g== 2024-09-16T05:45:36.907 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAAxudmhFEMNhAADcZ5FsWhkYDY1+jeW3527g=="}' 2024-09-16T05:45:36.908 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c6c76286-95d0-4a88-a09c-e0d1bfb2bae4 -i td/ceph-helpers/0/new.json 2024-09-16T05:45:37.201 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:45:37.211 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T05:45:37.212 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAAxudmhFEMNhAADcZ5FsWhkYDY1+jeW3527g== --osd-uuid c6c76286-95d0-4a88-a09c-e0d1bfb2bae4 2024-09-16T05:45:37.241 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:45:37.240+0000 7fa0c89dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:45:37.262 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:45:37.261+0000 7fa0c89dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:45:37.263 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:45:37.262+0000 7fa0c89dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:45:37.263 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:45:37.263+0000 7fa0c89dd640 -1 bdev(0x55b3a66abc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-16T05:45:37.264 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:45:37.263+0000 7fa0c89dd640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T05:45:43.608 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T05:45:43.608 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:45:43.609 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T05:45:43.609 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T05:45:43.609 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:45:44.132 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:45:44.133 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T05:45:44.133 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:45:44.133 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:45:44.133 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:45:44.133 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:45:44.159 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:45:44.158+0000 7f487a3c3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:45:44.192 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:45:44.191+0000 7f487a3c3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:45:44.193 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:45:44.192+0000 7f487a3c3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:45:44.502 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T05:45:44.503 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:45:44.503 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:45:44.503 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:45:44.503 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:45:44.503 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:45:44.503 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:45:44.503 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:45:44.504 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:45:44.504 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:45:44.913 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:45:45.297 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:45:45.297+0000 7f487a3c3640 -1 Falling back to public interface 2024-09-16T05:45:45.915 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:45:45.915 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:45:45.915 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:45:45.916 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:45:45.916 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:45:45.916 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:45:46.295 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:45:47.297 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:45:47.297 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:45:47.297 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:45:47.298 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:45:47.298 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:45:47.298 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:45:47.716 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:45:48.285 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:45:48.284+0000 7f487a3c3640 -1 osd.0 0 log_to_monitors true 2024-09-16T05:45:48.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:45:48.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:45:48.718 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:45:48.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:45:48.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:45:48.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:45:49.231 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:45:50.232 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:45:50.232 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:45:50.233 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:45:50.233 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:45:50.233 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:45:50.233 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:45:50.626 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:45:51.627 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:45:51.627 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:45:51.627 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:45:51.628 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:45:51.628 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:45:51.628 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:45:52.162 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:45:53.163 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:45:53.163 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:45:53.163 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:45:53.163 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:45:53.164 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:45:53.164 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:45:53.546 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:45:54.547 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:45:54.548 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:45:54.548 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:45:54.548 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:45:54.548 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:45:54.549 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:45:55.018 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:45:56.019 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:45:56.019 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:45:56.020 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:45:56.020 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:45:56.020 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:45:56.020 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:45:56.429 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:45:57.431 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:45:57.431 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:45:57.431 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:45:57.431 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:45:57.432 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:45:57.432 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:45:57.845 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:45:58.846 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:45:58.847 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:45:58.847 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:45:58.847 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:45:58.848 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:45:58.848 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:45:59.342 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:46:00.344 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:46:00.344 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:46:00.344 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:46:00.345 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:46:00.345 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:46:00.345 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:46:00.921 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:46:01.923 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:46:01.923 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:46:01.923 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:46:01.924 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:46:01.924 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:46:01.924 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:46:02.734 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:46:03.735 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:46:03.735 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:46:03.735 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:46:03.736 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:46:03.736 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:46:03.736 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:46:04.561 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:46:05.562 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:46:05.562 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:46:05.563 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:46:05.563 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:46:05.563 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:46:05.563 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:46:06.289 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:46:07.288 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:46:07.288 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:46:07.288 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:46:07.288 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:46:07.289 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:46:07.289 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:46:08.464 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:46:09.465 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:46:09.466 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:46:09.466 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:46:09.466 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:46:09.467 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:46:09.467 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:46:10.290 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:46:11.291 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:46:11.291 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:46:11.291 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:46:11.292 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:46:11.292 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:46:11.292 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:46:11.757 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/92711205,v1:127.0.0.1:6803/92711205] [v2:127.0.0.1:6804/92711205,v1:127.0.0.1:6805/92711205] exists,up c6c76286-95d0-4a88-a09c-e0d1bfb2bae4 2024-09-16T05:46:11.757 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:46:11.758 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:46:11.758 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:46:11.758 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1378: test_get_num_active_clean: create_rbd_pool 2024-09-16T05:46:11.758 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T05:46:12.115 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' does not exist 2024-09-16T05:46:12.126 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T05:46:12.126 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T05:46:12.587 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' already exists 2024-09-16T05:46:12.599 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T05:46:13.600 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T05:46:14.665 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1379: test_get_num_active_clean: wait_for_clean 2024-09-16T05:46:14.666 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T05:46:14.666 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T05:46:14.666 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T05:46:14.666 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T05:46:14.666 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T05:46:14.667 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T05:46:14.667 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T05:46:14.667 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T05:46:14.667 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T05:46:14.762 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T05:46:14.762 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T05:46:14.762 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T05:46:14.763 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T05:46:14.763 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-16T05:46:14.763 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-16T05:46:15.151 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-09-16T05:46:15.151 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-16T05:46:15.151 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:46:15.151 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T05:46:15.274 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836484 2024-09-16T05:46:15.274 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836484 2024-09-16T05:46:15.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836484' 2024-09-16T05:46:15.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:46:15.275 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836484 2024-09-16T05:46:15.275 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:46:15.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-16T05:46:15.276 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836484 2024-09-16T05:46:15.276 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:46:15.277 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836484 2024-09-16T05:46:15.277 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.0 seq 21474836484 2024-09-16T05:46:15.277 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2024-09-16T05:46:15.277 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:46:15.677 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836482 -lt 21474836484 2024-09-16T05:46:15.677 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-16T05:46:16.679 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T05:46:16.679 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:46:17.062 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836485 -lt 21474836484 2024-09-16T05:46:17.062 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T05:46:17.062 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:46:17.062 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:46:17.563 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T05:46:17.563 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T05:46:17.564 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T05:46:17.564 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T05:46:17.564 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T05:46:17.564 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T05:46:17.564 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T05:46:17.564 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T05:46:17.959 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T05:46:17.959 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T05:46:17.960 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:46:17.960 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:46:18.466 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T05:46:18.466 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T05:46:18.466 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T05:46:18.467 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1380: test_get_num_active_clean: get_num_active_clean 2024-09-16T05:46:18.467 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T05:46:18.467 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T05:46:18.467 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T05:46:18.467 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T05:46:18.467 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T05:46:18.869 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1380: test_get_num_active_clean: local num_active_clean=4 2024-09-16T05:46:18.869 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1381: test_get_num_active_clean: test 4 = 4 2024-09-16T05:46:18.869 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1382: test_get_num_active_clean: teardown td/ceph-helpers 2024-09-16T05:46:18.869 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:46:18.869 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:46:18.869 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:46:18.869 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:46:18.869 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:46:18.869 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:46:18.869 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:46:18.869 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:46:18.976 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:46:18.976 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:46:18.977 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:46:18.977 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:46:18.977 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:46:18.977 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:46:18.977 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:46:18.978 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:46:18.978 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:46:18.978 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:46:18.978 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:46:18.978 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:46:18.979 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:46:18.979 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:46:18.985 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:46:18.985 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:46:18.985 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:46:18.985 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:46:18.987 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:46:18.987 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:46:18.987 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T05:46:18.987 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_get_num_active_or_peered td/ceph-helpers 2024-09-16T05:46:18.987 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1402: test_get_num_active_or_peered: local dir=td/ceph-helpers 2024-09-16T05:46:18.987 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1404: test_get_num_active_or_peered: setup td/ceph-helpers 2024-09-16T05:46:18.987 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T05:46:18.987 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T05:46:18.987 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:46:18.987 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:46:18.988 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:46:18.988 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:46:18.988 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:46:18.988 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:46:18.988 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:46:18.988 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:46:18.988 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:46:18.988 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:46:18.989 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:46:18.989 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:46:18.990 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:46:18.990 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:46:18.990 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:46:18.991 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:46:18.991 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:46:18.991 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:46:18.991 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:46:18.992 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:46:18.993 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:46:18.993 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:46:18.994 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:46:18.994 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:46:18.994 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:46:18.994 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:46:18.995 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:46:18.995 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:46:18.995 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T05:46:18.996 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:46:18.996 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:46:18.996 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:46:18.996 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T05:46:18.998 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:46:18.998 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:46:18.998 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:46:18.998 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T05:46:18.998 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1405: test_get_num_active_or_peered: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:46:18.998 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T05:46:18.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:46:18.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:46:18.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:46:18.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T05:46:18.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:46:19.376 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:46:19.376 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:46:19.376 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:46:19.376 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:46:19.376 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:46:19.377 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:46:19.377 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:46:19.377 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:46:19.568 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:46:19.570 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:46:19.570 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:46:19.570 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:46:19.570 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:46:19.570 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:46:19.570 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:46:19.570 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:46:19.570 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:46:19.570 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:46:19.571 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:46:19.571 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:46:19.571 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:46:19.571 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:46:19.571 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T05:46:19.649 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:46:19.649 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:46:19.649 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:46:19.650 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:46:19.650 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:46:19.650 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:46:19.650 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:46:19.650 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:46:19.650 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:46:19.650 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:46:19.650 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:46:19.650 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:46:19.650 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:46:19.650 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T05:46:19.727 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1406: test_get_num_active_or_peered: run_mgr td/ceph-helpers x 2024-09-16T05:46:19.727 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T05:46:19.727 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T05:46:19.727 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T05:46:19.727 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T05:46:19.728 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T05:46:19.728 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T05:46:20.573 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T05:46:20.573 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:46:20.573 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:46:20.573 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:46:20.574 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:46:20.574 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:46:20.574 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:46:20.574 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:46:20.574 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:46:20.605 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1407: test_get_num_active_or_peered: run_osd td/ceph-helpers 0 2024-09-16T05:46:20.605 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T05:46:20.605 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:46:20.605 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T05:46:20.605 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:46:20.606 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T05:46:20.606 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T05:46:20.606 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:46:20.606 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:46:20.606 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:46:20.606 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T05:46:20.606 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T05:46:20.606 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:46:20.606 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:46:20.606 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T05:46:20.606 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:46:20.607 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:46:20.607 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:46:20.607 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:46:20.607 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:46:20.607 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:46:20.607 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:46:20.607 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:46:20.607 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:46:20.607 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:46:20.607 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:46:20.608 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T05:46:20.608 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T05:46:20.608 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:46:20.608 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:46:20.608 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:46:20.608 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:46:20.608 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:46:20.608 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:46:20.608 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T05:46:20.608 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:46:20.608 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9befaefa-0a77-445e-bea9-f359ff30f464 2024-09-16T05:46:20.608 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9befaefa-0a77-445e-bea9-f359ff30f464' 2024-09-16T05:46:20.609 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 9befaefa-0a77-445e-bea9-f359ff30f464 2024-09-16T05:46:20.609 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:46:20.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAsxudmEMImJRAA10mFBnlUmmLuIqAJv1l1LQ== 2024-09-16T05:46:20.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAsxudmEMImJRAA10mFBnlUmmLuIqAJv1l1LQ=="}' 2024-09-16T05:46:20.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9befaefa-0a77-445e-bea9-f359ff30f464 -i td/ceph-helpers/0/new.json 2024-09-16T05:46:20.909 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:46:20.919 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T05:46:20.920 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAsxudmEMImJRAA10mFBnlUmmLuIqAJv1l1LQ== --osd-uuid 9befaefa-0a77-445e-bea9-f359ff30f464 2024-09-16T05:46:20.949 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:46:20.948+0000 7f3312614640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:46:20.978 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:46:20.978+0000 7f3312614640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:46:20.979 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:46:20.979+0000 7f3312614640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:46:20.980 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:46:20.979+0000 7f3312614640 -1 bdev(0x56003ee25c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-16T05:46:20.980 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:46:20.979+0000 7f3312614640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T05:46:26.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T05:46:26.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:46:26.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T05:46:26.401 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T05:46:26.402 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:46:26.922 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:46:26.923 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T05:46:26.923 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:46:26.923 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:46:26.923 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:46:26.923 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:46:26.949 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:46:26.948+0000 7f1fc95dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:46:26.964 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:46:26.964+0000 7f1fc95dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:46:26.965 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:46:26.965+0000 7f1fc95dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:46:27.369 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T05:46:27.369 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:46:27.369 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:46:27.369 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:46:27.369 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:46:27.369 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:46:27.370 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:46:27.370 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:46:27.370 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:46:27.370 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:46:27.778 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:46:28.308 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:46:28.308+0000 7f1fc95dd640 -1 Falling back to public interface 2024-09-16T05:46:28.779 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:46:28.780 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:46:28.780 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:46:28.781 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:46:28.781 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:46:28.781 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:46:29.195 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:46:30.196 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:46:30.197 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:46:30.197 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:46:30.198 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:46:30.198 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:46:30.198 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:46:30.580 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:46:31.116 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:46:31.116+0000 7f1fc95dd640 -1 osd.0 0 log_to_monitors true 2024-09-16T05:46:31.581 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:46:31.581 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:46:31.582 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:46:31.582 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:46:31.582 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:46:31.582 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:46:32.028 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:46:33.029 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:46:33.029 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:46:33.029 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:46:33.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:46:33.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:46:33.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:46:33.424 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:46:34.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:46:34.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:46:34.426 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:46:34.426 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:46:34.426 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:46:34.427 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:46:34.890 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:46:35.891 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:46:35.891 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:46:35.891 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:46:35.892 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:46:35.892 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:46:35.892 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:46:36.346 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:46:37.347 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:46:37.347 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:46:37.348 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:46:37.348 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:46:37.348 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:46:37.348 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:46:37.850 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:46:38.851 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:46:38.852 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:46:38.852 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:46:38.852 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:46:38.853 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:46:38.853 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:46:39.254 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:46:40.255 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:46:40.255 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:46:40.256 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:46:40.256 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:46:40.256 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:46:40.256 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:46:40.652 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:46:41.651 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:46:41.652 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:46:41.652 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:46:41.652 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:46:41.652 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:46:41.652 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:46:42.038 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:46:43.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:46:43.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:46:43.040 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:46:43.040 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:46:43.040 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:46:43.040 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:46:43.904 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:46:44.906 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:46:44.906 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:46:44.906 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:46:44.907 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:46:44.907 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:46:44.907 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:46:45.740 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:46:46.742 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:46:46.742 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:46:46.742 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:46:46.743 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:46:46.743 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:46:46.743 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:46:47.422 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:46:48.423 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:46:48.424 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:46:48.424 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:46:48.424 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:46:48.424 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:46:48.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:46:48.952 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:46:49.954 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:46:49.954 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:46:49.954 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:46:49.955 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:46:49.955 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:46:49.955 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:46:50.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:46:51.786 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:46:51.786 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:46:51.786 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:46:51.787 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:46:51.787 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:46:51.787 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:46:52.588 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:46:53.589 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:46:53.590 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:46:53.590 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:46:53.590 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:46:53.590 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:46:53.590 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:46:54.009 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/254597120,v1:127.0.0.1:6803/254597120] [v2:127.0.0.1:6804/254597120,v1:127.0.0.1:6805/254597120] exists,up 9befaefa-0a77-445e-bea9-f359ff30f464 2024-09-16T05:46:54.010 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:46:54.010 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:46:54.010 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:46:54.010 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1408: test_get_num_active_or_peered: create_rbd_pool 2024-09-16T05:46:54.010 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T05:46:54.419 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' does not exist 2024-09-16T05:46:54.430 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T05:46:54.430 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T05:46:54.936 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' already exists 2024-09-16T05:46:54.947 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T05:46:55.949 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T05:46:57.080 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1409: test_get_num_active_or_peered: wait_for_clean 2024-09-16T05:46:57.080 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T05:46:57.080 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T05:46:57.080 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T05:46:57.081 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T05:46:57.081 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T05:46:57.081 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T05:46:57.081 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T05:46:57.081 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T05:46:57.081 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T05:46:57.171 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T05:46:57.171 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T05:46:57.172 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T05:46:57.172 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T05:46:57.172 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-16T05:46:57.172 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-16T05:46:57.581 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-09-16T05:46:57.581 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-16T05:46:57.582 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:46:57.582 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T05:46:57.695 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836490 2024-09-16T05:46:57.695 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836490 2024-09-16T05:46:57.696 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836490' 2024-09-16T05:46:57.696 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:46:57.696 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836490 2024-09-16T05:46:57.696 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:46:57.696 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-16T05:46:57.697 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836490 2024-09-16T05:46:57.697 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:46:57.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836490 2024-09-16T05:46:57.698 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.0 seq 21474836490 2024-09-16T05:46:57.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2024-09-16T05:46:57.698 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:46:58.069 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836488 -lt 21474836490 2024-09-16T05:46:58.069 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-16T05:46:59.070 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T05:46:59.071 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:46:59.456 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836490 -lt 21474836490 2024-09-16T05:46:59.456 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T05:46:59.456 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:46:59.456 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:46:59.985 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T05:46:59.985 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T05:46:59.985 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T05:46:59.985 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T05:46:59.985 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T05:46:59.985 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T05:46:59.986 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T05:46:59.986 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T05:47:00.367 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T05:47:00.368 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T05:47:00.368 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:47:00.368 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:47:00.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T05:47:00.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T05:47:00.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T05:47:00.896 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1410: test_get_num_active_or_peered: get_num_active_or_peered 2024-09-16T05:47:00.897 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1394: get_num_active_or_peered: local expression 2024-09-16T05:47:00.897 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-09-16T05:47:00.897 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-09-16T05:47:00.897 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: get_num_active_or_peered: ceph --format json pg dump pgs 2024-09-16T05:47:00.897 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-09-16T05:47:01.298 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1410: test_get_num_active_or_peered: local num_peered=4 2024-09-16T05:47:01.298 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1411: test_get_num_active_or_peered: test 4 = 4 2024-09-16T05:47:01.299 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1412: test_get_num_active_or_peered: teardown td/ceph-helpers 2024-09-16T05:47:01.299 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:47:01.299 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:47:01.299 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:47:01.299 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:47:01.299 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:47:01.299 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:47:01.299 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:47:01.299 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:47:01.404 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:47:01.404 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:47:01.405 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:47:01.405 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:47:01.406 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:47:01.406 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:47:01.406 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:47:01.407 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:47:01.407 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:47:01.408 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:47:01.408 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:47:01.408 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:47:01.409 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:47:01.409 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:47:01.415 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:47:01.415 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:47:01.415 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:47:01.415 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:47:01.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:47:01.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:47:01.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T05:47:01.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_get_num_pgs td/ceph-helpers 2024-09-16T05:47:01.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1429: test_get_num_pgs: local dir=td/ceph-helpers 2024-09-16T05:47:01.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1431: test_get_num_pgs: setup td/ceph-helpers 2024-09-16T05:47:01.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T05:47:01.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T05:47:01.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:47:01.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:47:01.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:47:01.419 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:47:01.419 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:47:01.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:47:01.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:47:01.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:47:01.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:47:01.420 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:47:01.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:47:01.420 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:47:01.421 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:47:01.421 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:47:01.421 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:47:01.422 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:47:01.422 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:47:01.422 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:47:01.422 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:47:01.423 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:47:01.424 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:47:01.424 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:47:01.425 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:47:01.425 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:47:01.425 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:47:01.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:47:01.426 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:47:01.426 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:47:01.426 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T05:47:01.427 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:47:01.427 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:47:01.427 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:47:01.427 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T05:47:01.429 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:47:01.429 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:47:01.429 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:47:01.429 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T05:47:01.430 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: test_get_num_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:47:01.430 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T05:47:01.430 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:47:01.430 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:47:01.430 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:47:01.430 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T05:47:01.430 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:47:01.808 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:47:01.809 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:47:01.809 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:47:01.809 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:47:01.809 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:47:01.809 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:47:01.809 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:47:01.809 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:47:02.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:47:02.090 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:47:02.090 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:47:02.090 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:47:02.091 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:47:02.091 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:47:02.091 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:47:02.091 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:47:02.091 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:47:02.091 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:47:02.091 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:47:02.091 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:47:02.091 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:47:02.091 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:47:02.091 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T05:47:02.092 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:47:02.092 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:47:02.092 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:47:02.092 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:47:02.092 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:47:02.092 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:47:02.092 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:47:02.092 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:47:02.092 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:47:02.092 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:47:02.092 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:47:02.093 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:47:02.093 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:47:02.093 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T05:47:02.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: test_get_num_pgs: run_mgr td/ceph-helpers x 2024-09-16T05:47:02.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T05:47:02.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T05:47:02.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T05:47:02.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T05:47:02.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T05:47:02.160 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T05:47:03.006 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T05:47:03.006 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:47:03.006 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:47:03.006 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:47:03.006 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:47:03.006 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:47:03.006 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:47:03.007 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:47:03.007 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:47:03.038 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: test_get_num_pgs: run_osd td/ceph-helpers 0 2024-09-16T05:47:03.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T05:47:03.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:47:03.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T05:47:03.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:47:03.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T05:47:03.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T05:47:03.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:47:03.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:47:03.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:47:03.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T05:47:03.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T05:47:03.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:47:03.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:47:03.040 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T05:47:03.040 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:47:03.040 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:47:03.040 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:47:03.040 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:47:03.040 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:47:03.040 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:47:03.041 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:47:03.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:47:03.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:47:03.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:47:03.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:47:03.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T05:47:03.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T05:47:03.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:47:03.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:47:03.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:47:03.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:47:03.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:47:03.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:47:03.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T05:47:03.042 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:47:03.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fad80ece-5bf4-4b97-bef5-12649dccad6f 2024-09-16T05:47:03.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 fad80ece-5bf4-4b97-bef5-12649dccad6f' 2024-09-16T05:47:03.043 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 fad80ece-5bf4-4b97-bef5-12649dccad6f 2024-09-16T05:47:03.043 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:47:03.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBXxudmaU5oAxAADxa/M1243Z86ZyHThiTkcQ== 2024-09-16T05:47:03.058 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBXxudmaU5oAxAADxa/M1243Z86ZyHThiTkcQ=="}' 2024-09-16T05:47:03.058 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fad80ece-5bf4-4b97-bef5-12649dccad6f -i td/ceph-helpers/0/new.json 2024-09-16T05:47:03.316 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:47:03.326 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T05:47:03.327 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBXxudmaU5oAxAADxa/M1243Z86ZyHThiTkcQ== --osd-uuid fad80ece-5bf4-4b97-bef5-12649dccad6f 2024-09-16T05:47:03.356 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:47:03.356+0000 7fb03620c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:47:03.385 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:47:03.385+0000 7fb03620c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:47:03.386 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:47:03.386+0000 7fb03620c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:47:03.387 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:47:03.386+0000 7fb03620c640 -1 bdev(0x559ea6f2bc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-16T05:47:03.387 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:47:03.386+0000 7fb03620c640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T05:47:08.963 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T05:47:08.963 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:47:08.964 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T05:47:08.964 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T05:47:08.964 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:47:09.513 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:47:09.514 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T05:47:09.514 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:47:09.514 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:47:09.514 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:47:09.514 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:47:09.540 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:47:09.539+0000 7f599da0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:47:09.542 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:47:09.541+0000 7f599da0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:47:09.543 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:47:09.542+0000 7f599da0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:47:09.928 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T05:47:09.928 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:47:09.929 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:47:09.929 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:47:09.929 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:47:09.929 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:47:09.929 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:47:09.929 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:47:09.930 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:47:09.930 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:47:10.337 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:47:11.139 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:47:11.139+0000 7f599da0d640 -1 Falling back to public interface 2024-09-16T05:47:11.339 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:47:11.339 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:47:11.339 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:47:11.339 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:47:11.339 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:47:11.339 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:47:11.716 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:47:12.717 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:47:12.717 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:47:12.717 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:47:12.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:47:12.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:47:12.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:47:13.093 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:47:13.817 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:47:13.816+0000 7f599da0d640 -1 osd.0 0 log_to_monitors true 2024-09-16T05:47:14.095 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:47:14.095 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:47:14.095 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:47:14.096 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:47:14.096 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:47:14.096 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:47:14.568 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:47:15.569 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:47:15.570 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:47:15.570 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:47:15.570 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:47:15.570 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:47:15.570 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:47:15.961 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:47:16.962 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:47:16.962 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:47:16.962 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:47:16.962 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:47:16.963 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:47:16.963 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:47:17.362 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:47:18.364 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:47:18.364 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:47:18.364 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:47:18.365 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:47:18.365 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:47:18.366 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:47:18.747 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:47:19.748 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:47:19.748 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:47:19.748 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:47:19.749 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:47:19.749 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:47:19.749 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:47:20.156 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:47:21.157 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:47:21.157 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:47:21.158 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:47:21.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:47:21.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:47:21.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:47:21.603 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:47:22.604 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:47:22.604 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:47:22.605 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:47:22.605 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:47:22.605 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:47:22.605 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:47:23.025 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:47:24.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:47:24.027 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:47:24.027 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:47:24.027 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:47:24.027 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:47:24.027 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:47:24.409 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:47:25.410 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:47:25.410 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:47:25.411 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:47:25.411 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:47:25.411 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:47:25.411 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:47:26.171 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:47:27.172 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:47:27.172 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:47:27.173 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:47:27.173 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:47:27.173 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:47:27.173 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:47:28.054 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:47:29.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:47:29.056 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:47:29.056 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:47:29.056 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:47:29.056 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:47:29.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:47:29.893 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:47:30.895 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:47:30.895 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:47:30.895 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:47:30.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:47:30.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:47:30.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:47:31.359 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:47:32.361 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:47:32.361 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:47:32.361 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:47:32.362 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:47:32.362 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:47:32.362 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:47:33.114 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:47:34.115 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:47:34.116 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:47:34.116 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:47:34.116 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:47:34.116 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:47:34.116 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:47:34.884 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:47:35.886 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:47:35.886 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:47:35.887 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:47:35.887 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:47:35.887 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:47:35.887 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:47:36.616 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:47:37.618 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:47:37.618 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:47:37.618 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T05:47:37.618 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T05:47:37.619 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:47:37.619 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:47:38.010 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3052523882,v1:127.0.0.1:6803/3052523882] [v2:127.0.0.1:6804/3052523882,v1:127.0.0.1:6805/3052523882] exists,up fad80ece-5bf4-4b97-bef5-12649dccad6f 2024-09-16T05:47:38.010 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:47:38.010 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:47:38.010 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:47:38.011 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: test_get_num_pgs: create_rbd_pool 2024-09-16T05:47:38.011 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T05:47:38.412 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' does not exist 2024-09-16T05:47:38.422 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T05:47:38.423 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T05:47:38.888 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' already exists 2024-09-16T05:47:38.899 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T05:47:39.901 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T05:47:40.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1436: test_get_num_pgs: wait_for_clean 2024-09-16T05:47:40.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T05:47:40.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T05:47:40.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T05:47:40.933 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T05:47:40.934 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T05:47:40.934 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T05:47:40.934 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T05:47:40.934 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T05:47:40.934 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T05:47:41.029 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T05:47:41.029 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T05:47:41.029 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T05:47:41.029 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T05:47:41.029 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-16T05:47:41.029 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-16T05:47:41.429 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-09-16T05:47:41.429 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-16T05:47:41.429 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:47:41.429 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T05:47:41.540 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836487 2024-09-16T05:47:41.540 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836487 2024-09-16T05:47:41.540 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487' 2024-09-16T05:47:41.540 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:47:41.541 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836487 2024-09-16T05:47:41.541 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:47:41.541 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-16T05:47:41.542 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836487 2024-09-16T05:47:41.542 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:47:41.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836487 2024-09-16T05:47:41.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-09-16T05:47:41.543 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.0 seq 21474836487 2024-09-16T05:47:41.543 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:47:41.912 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836485 -lt 21474836487 2024-09-16T05:47:41.912 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-16T05:47:42.913 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T05:47:42.914 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:47:43.313 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836488 -lt 21474836487 2024-09-16T05:47:43.313 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T05:47:43.314 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:47:43.314 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:47:43.816 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T05:47:43.816 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T05:47:43.816 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T05:47:43.816 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T05:47:43.816 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T05:47:43.816 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T05:47:43.816 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T05:47:43.816 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T05:47:44.218 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T05:47:44.219 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T05:47:44.219 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:47:44.219 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:47:44.716 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T05:47:44.716 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T05:47:44.716 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T05:47:44.716 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1437: test_get_num_pgs: get_num_pgs 2024-09-16T05:47:44.716 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:47:44.716 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:47:45.224 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1437: test_get_num_pgs: local num_pgs=4 2024-09-16T05:47:45.224 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1438: test_get_num_pgs: test 4 -gt 0 2024-09-16T05:47:45.224 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1439: test_get_num_pgs: teardown td/ceph-helpers 2024-09-16T05:47:45.224 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:47:45.225 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:47:45.225 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:47:45.225 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:47:45.225 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:47:45.225 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:47:45.225 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:47:45.225 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:47:45.330 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:47:45.330 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:47:45.330 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:47:45.331 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:47:45.332 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:47:45.332 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:47:45.332 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:47:45.333 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:47:45.333 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:47:45.333 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:47:45.333 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:47:45.334 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:47:45.335 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:47:45.335 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:47:45.341 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:47:45.341 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:47:45.341 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:47:45.341 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:47:45.343 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:47:45.343 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:47:45.343 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T05:47:45.344 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_get_op_scheduler td/ceph-helpers 2024-09-16T05:47:45.344 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: test_get_op_scheduler: local dir=td/ceph-helpers 2024-09-16T05:47:45.344 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: test_get_op_scheduler: setup td/ceph-helpers 2024-09-16T05:47:45.344 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T05:47:45.345 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T05:47:45.345 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:47:45.345 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:47:45.345 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:47:45.345 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:47:45.345 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:47:45.345 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:47:45.345 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:47:45.345 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:47:45.348 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:47:45.348 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:47:45.348 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:47:45.348 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:47:45.348 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:47:45.348 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:47:45.348 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:47:45.349 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:47:45.349 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:47:45.349 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:47:45.349 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:47:45.350 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:47:45.350 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:47:45.350 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:47:45.351 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:47:45.351 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:47:45.351 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:47:45.351 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:47:45.351 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:47:45.351 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:47:45.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T05:47:45.352 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:47:45.352 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:47:45.353 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:47:45.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T05:47:45.354 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:47:45.354 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:47:45.355 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:47:45.355 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T05:47:45.355 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: test_get_op_scheduler: run_mon td/ceph-helpers a 2024-09-16T05:47:45.355 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T05:47:45.355 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:47:45.355 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:47:45.355 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:47:45.355 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T05:47:45.355 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-16T05:47:45.721 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:47:45.721 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:47:45.722 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:47:45.722 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:47:45.722 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:47:45.722 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:47:45.722 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:47:45.722 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T05:47:45.903 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:47:45.904 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:47:45.904 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:47:45.904 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:47:45.904 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:47:45.904 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:47:45.904 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:47:45.905 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:47:45.905 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:47:45.905 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:47:45.906 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:47:45.906 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:47:45.906 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:47:45.906 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:47:45.906 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T05:47:45.985 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:47:45.985 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:47:45.985 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:47:45.985 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:47:45.985 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:47:45.985 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:47:45.985 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:47:45.986 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:47:45.986 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:47:45.986 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:47:45.986 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:47:45.986 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:47:45.986 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:47:45.986 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T05:47:46.063 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: test_get_op_scheduler: run_mgr td/ceph-helpers x 2024-09-16T05:47:46.063 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T05:47:46.063 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T05:47:46.064 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T05:47:46.064 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T05:47:46.064 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T05:47:46.064 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T05:47:46.900 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T05:47:46.900 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:47:46.900 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:47:46.900 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:47:46.900 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:47:46.900 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:47:46.900 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:47:46.901 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:47:46.901 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:47:46.931 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: test_get_op_scheduler: run_osd td/ceph-helpers 0 --osd_op_queue=wpq 2024-09-16T05:47:46.931 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T05:47:46.931 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:47:46.931 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T05:47:46.931 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:47:46.932 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T05:47:46.932 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T05:47:46.932 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:47:46.932 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:47:46.932 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:47:46.932 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T05:47:46.932 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T05:47:46.932 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:47:46.932 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:47:46.932 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T05:47:46.932 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:47:46.932 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:47:46.932 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:47:46.933 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:47:46.933 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:47:46.933 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:47:46.933 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:47:46.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:47:46.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:47:46.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:47:46.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:47:46.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T05:47:46.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T05:47:46.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:47:46.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:47:46.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:47:46.934 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:47:46.934 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:47:46.934 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=wpq 2024-09-16T05:47:46.934 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T05:47:46.934 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:47:46.934 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0a0bccf8-8359-48ef-9c37-7ed12eac8d3c 2024-09-16T05:47:46.934 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0a0bccf8-8359-48ef-9c37-7ed12eac8d3c' 2024-09-16T05:47:46.935 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 0a0bccf8-8359-48ef-9c37-7ed12eac8d3c 2024-09-16T05:47:46.935 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:47:46.951 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCCxudmirunOBAAsUGHW0rA8moqb39jALONog== 2024-09-16T05:47:46.951 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCCxudmirunOBAAsUGHW0rA8moqb39jALONog=="}' 2024-09-16T05:47:46.951 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0a0bccf8-8359-48ef-9c37-7ed12eac8d3c -i td/ceph-helpers/0/new.json 2024-09-16T05:47:47.244 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:47:47.254 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T05:47:47.255 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=wpq --mkfs --key AQCCxudmirunOBAAsUGHW0rA8moqb39jALONog== --osd-uuid 0a0bccf8-8359-48ef-9c37-7ed12eac8d3c 2024-09-16T05:47:47.285 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:47:47.283+0000 7fb7c5943640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:47:47.313 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:47:47.312+0000 7fb7c5943640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:47:47.315 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:47:47.313+0000 7fb7c5943640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:47:47.315 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:47:47.314+0000 7fb7c5943640 -1 bdev(0x5595a5a15c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-16T05:47:47.315 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:47:47.314+0000 7fb7c5943640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T05:47:53.072 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T05:47:53.072 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:47:53.072 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T05:47:53.073 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T05:47:53.073 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:47:53.586 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:47:53.587 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T05:47:53.587 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=wpq 2024-09-16T05:47:53.587 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:47:53.587 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:47:53.587 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:47:53.613 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:47:53.613+0000 7f86e8c44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:47:53.632 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:47:53.632+0000 7f86e8c44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:47:53.633 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:47:53.633+0000 7f86e8c44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:47:54.029 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T05:47:54.029 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:47:54.029 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:47:54.029 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:47:54.029 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:47:54.029 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:47:54.029 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:47:54.029 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:47:54.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:47:54.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:47:54.397 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:47:55.222 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:47:55.222+0000 7f86e8c44640 -1 Falling back to public interface 2024-09-16T05:47:55.398 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:47:55.399 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:47:55.399 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:47:55.399 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:47:55.399 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:47:55.399 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:47:55.775 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:47:56.777 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:47:56.777 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:47:56.777 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:47:56.778 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:47:56.778 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:47:56.778 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:47:57.165 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:47:57.752 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:47:57.751+0000 7f86e8c44640 -1 osd.0 0 log_to_monitors true 2024-09-16T05:47:58.166 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:47:58.235 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:47:58.235 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:47:58.236 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:47:58.236 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:47:58.236 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:47:58.541 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1278717475,v1:127.0.0.1:6803/1278717475] [v2:127.0.0.1:6804/1278717475,v1:127.0.0.1:6805/1278717475] exists,up 0a0bccf8-8359-48ef-9c37-7ed12eac8d3c 2024-09-16T05:47:58.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:47:58.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:47:58.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:47:58.543 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: test_get_op_scheduler: get_op_scheduler 0 2024-09-16T05:47:58.543 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2254: get_op_scheduler: local id=0 2024-09-16T05:47:58.543 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: get_op_scheduler: get_config osd 0 osd_op_queue 2024-09-16T05:47:58.544 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=osd 2024-09-16T05:47:58.544 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=0 2024-09-16T05:47:58.544 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_op_queue 2024-09-16T05:47:58.544 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_op_queue 2024-09-16T05:47:58.544 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path osd.0 2024-09-16T05:47:58.544 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T05:47:58.544 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T05:47:58.544 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:47:58.544 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:47:58.544 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:47:58.544 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-osd.0.asok 2024-09-16T05:47:58.545 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:47:58.545 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-osd.0.asok config get osd_op_queue 2024-09-16T05:47:58.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: test_get_op_scheduler: test wpq = wpq 2024-09-16T05:47:58.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: test_get_op_scheduler: run_osd td/ceph-helpers 1 --osd_op_queue=mclock_scheduler 2024-09-16T05:47:58.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T05:47:58.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:47:58.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T05:47:58.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:47:58.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-09-16T05:47:58.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T05:47:58.633 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:47:58.633 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:47:58.633 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:47:58.633 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-16T05:47:58.633 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-16T05:47:58.633 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:47:58.633 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:47:58.633 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T05:47:58.633 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:47:58.633 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:47:58.633 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:47:58.633 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:47:58.633 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:47:58.634 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:47:58.634 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:47:58.634 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:47:58.634 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:47:58.634 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:47:58.634 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:47:58.634 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T05:47:58.634 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T05:47:58.634 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:47:58.634 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:47:58.634 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:47:58.634 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:47:58.634 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:47:58.634 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-16T05:47:58.635 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-09-16T05:47:58.635 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:47:58.635 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8fb0eb74-2079-47de-8b58-6266a6ac92bb 2024-09-16T05:47:58.635 INFO:tasks.workunit.client.0.smithi178.stdout:add osd1 8fb0eb74-2079-47de-8b58-6266a6ac92bb 2024-09-16T05:47:58.635 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 8fb0eb74-2079-47de-8b58-6266a6ac92bb' 2024-09-16T05:47:58.635 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:47:58.650 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCOxudm3EvBJhAAljpmgwIXtvM21RYbUadRDQ== 2024-09-16T05:47:58.651 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCOxudm3EvBJhAAljpmgwIXtvM21RYbUadRDQ=="}' 2024-09-16T05:47:58.651 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8fb0eb74-2079-47de-8b58-6266a6ac92bb -i td/ceph-helpers/1/new.json 2024-09-16T05:47:59.116 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:47:59.127 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-09-16T05:47:59.128 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQCOxudm3EvBJhAAljpmgwIXtvM21RYbUadRDQ== --osd-uuid 8fb0eb74-2079-47de-8b58-6266a6ac92bb 2024-09-16T05:47:59.157 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:47:59.156+0000 7f3027ce0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:47:59.204 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:47:59.203+0000 7f3027ce0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:47:59.204 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:47:59.204+0000 7f3027ce0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:47:59.205 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:47:59.204+0000 7f3027ce0640 -1 bdev(0x561b43b9bc00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2024-09-16T05:47:59.205 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:47:59.205+0000 7f3027ce0640 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-16T05:48:05.442 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-16T05:48:05.442 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:48:05.443 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd1 key to auth repository 2024-09-16T05:48:05.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T05:48:05.444 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:48:05.957 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.1 2024-09-16T05:48:05.958 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T05:48:05.958 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-16T05:48:05.958 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:48:05.958 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:48:05.958 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:48:05.985 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:48:05.984+0000 7f722bbfd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:48:06.016 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:48:06.016+0000 7f722bbfd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:48:06.017 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:48:06.017+0000 7f722bbfd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:48:06.371 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T05:48:06.371 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:48:06.372 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T05:48:06.372 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:48:06.372 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:48:06.372 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:48:06.372 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:48:06.373 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:48:06.373 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:48:06.373 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:48:06.744 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:48:07.595 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:48:07.595+0000 7f722bbfd640 -1 Falling back to public interface 2024-09-16T05:48:07.746 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:48:07.746 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:48:07.746 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:48:07.746 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:48:07.747 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:48:07.747 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:48:08.123 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:48:09.124 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:48:09.125 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:48:09.125 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:48:09.125 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:48:09.125 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:48:09.125 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:48:09.542 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:48:10.512 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:48:10.512+0000 7f722bbfd640 -1 osd.1 0 log_to_monitors true 2024-09-16T05:48:10.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:48:10.544 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:48:10.544 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:48:10.545 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:48:10.545 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:48:10.545 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:48:11.060 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:48:12.061 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:48:12.062 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:48:12.062 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:48:12.062 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:48:12.062 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:48:12.063 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:48:12.501 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:48:13.502 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:48:13.503 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:48:13.503 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:48:13.503 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:48:13.503 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:48:13.503 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:48:13.907 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:48:14.908 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:48:14.908 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:48:14.909 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:48:14.909 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:48:14.909 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:48:14.909 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:48:15.388 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:48:16.389 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:48:16.390 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:48:16.390 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:48:16.390 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:48:16.390 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:48:16.390 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:48:16.812 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:48:17.813 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:48:17.813 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:48:17.813 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:48:17.814 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:48:17.814 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:48:17.814 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:48:18.192 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:48:19.194 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:48:19.194 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:48:19.194 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:48:19.195 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:48:19.195 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:48:19.195 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:48:19.669 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:48:20.671 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:48:20.671 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:48:20.671 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:48:20.672 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:48:20.672 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:48:20.672 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:48:21.094 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:48:22.095 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:48:22.095 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:48:22.095 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:48:22.096 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:48:22.096 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:48:22.096 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:48:22.489 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:48:23.491 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:48:23.491 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:48:23.491 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:48:23.491 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:48:23.492 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:48:23.492 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:48:24.503 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:48:25.504 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:48:25.504 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:48:25.505 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:48:25.505 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:48:25.505 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:48:25.506 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:48:26.255 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:48:27.256 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:48:27.257 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:48:27.257 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:48:27.257 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:48:27.257 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:48:27.257 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:48:28.113 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:48:29.114 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:48:29.114 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:48:29.114 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:48:29.115 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:48:29.115 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:48:29.115 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:48:29.820 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:48:30.821 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:48:30.821 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:48:30.822 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:48:30.822 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:48:30.822 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:48:30.822 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:48:31.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:48:32.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:48:32.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:48:32.484 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:48:32.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:48:32.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:48:32.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:48:32.934 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:48:33.936 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:48:33.936 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:48:33.936 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T05:48:33.937 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T05:48:33.937 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:48:33.937 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:48:34.354 INFO:tasks.workunit.client.0.smithi178.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2527340101,v1:127.0.0.1:6811/2527340101] [v2:127.0.0.1:6812/2527340101,v1:127.0.0.1:6813/2527340101] exists,up 8fb0eb74-2079-47de-8b58-6266a6ac92bb 2024-09-16T05:48:34.357 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:48:34.357 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:48:34.357 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:48:34.358 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: test_get_op_scheduler: get_op_scheduler 1 2024-09-16T05:48:34.358 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2254: get_op_scheduler: local id=1 2024-09-16T05:48:34.358 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: get_op_scheduler: get_config osd 1 osd_op_queue 2024-09-16T05:48:34.358 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=osd 2024-09-16T05:48:34.358 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=1 2024-09-16T05:48:34.358 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_op_queue 2024-09-16T05:48:34.358 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_op_queue 2024-09-16T05:48:34.358 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path osd.1 2024-09-16T05:48:34.358 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T05:48:34.359 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T05:48:34.359 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:48:34.359 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:48:34.359 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:48:34.359 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-osd.1.asok 2024-09-16T05:48:34.359 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:48:34.359 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-osd.1.asok config get osd_op_queue 2024-09-16T05:48:34.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: test_get_op_scheduler: test mclock_scheduler = mclock_scheduler 2024-09-16T05:48:34.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: test_get_op_scheduler: teardown td/ceph-helpers 2024-09-16T05:48:34.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:48:34.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:48:34.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:48:34.441 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:48:34.441 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:48:34.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:48:34.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:48:34.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:48:34.546 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:48:34.547 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:48:34.547 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:48:34.548 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:48:34.549 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:48:34.549 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:48:34.549 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:48:34.550 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:48:34.550 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:48:34.550 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:48:34.550 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:48:34.551 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:48:34.552 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:48:34.552 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:48:34.558 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:48:34.558 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:48:34.558 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:48:34.558 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:48:34.560 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:48:34.560 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:48:34.560 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T05:48:34.560 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_get_osd_id_used_by_pgs td/ceph-helpers 2024-09-16T05:48:34.560 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1459: test_get_osd_id_used_by_pgs: local dir=td/ceph-helpers 2024-09-16T05:48:34.560 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1461: test_get_osd_id_used_by_pgs: setup td/ceph-helpers 2024-09-16T05:48:34.561 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T05:48:34.561 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T05:48:34.561 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:48:34.561 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:48:34.561 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:48:34.561 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:48:34.561 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:48:34.561 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:48:34.561 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:48:34.561 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:48:34.562 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:48:34.562 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:48:34.562 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:48:34.562 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:48:34.563 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:48:34.563 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:48:34.563 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:48:34.564 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:48:34.564 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:48:34.564 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:48:34.565 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:48:34.565 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:48:34.566 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:48:34.566 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:48:34.567 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:48:34.567 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:48:34.567 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:48:34.567 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:48:34.568 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:48:34.568 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:48:34.568 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T05:48:34.569 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:48:34.569 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:48:34.569 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:48:34.569 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T05:48:34.571 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:48:34.571 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:48:34.571 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:48:34.572 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T05:48:34.572 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1462: test_get_osd_id_used_by_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:48:34.572 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T05:48:34.572 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:48:34.572 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:48:34.572 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:48:34.572 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T05:48:34.572 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:48:34.948 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:48:34.949 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:48:34.949 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:48:34.949 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:48:34.949 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:48:34.949 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:48:34.949 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:48:34.949 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:48:35.166 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:48:35.167 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:48:35.167 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:48:35.167 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:48:35.167 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:48:35.168 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:48:35.168 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:48:35.168 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:48:35.168 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:48:35.168 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:48:35.168 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:48:35.168 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:48:35.168 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:48:35.168 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:48:35.169 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T05:48:35.248 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:48:35.248 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:48:35.248 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:48:35.248 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:48:35.248 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:48:35.248 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:48:35.249 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:48:35.249 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:48:35.249 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:48:35.249 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:48:35.249 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:48:35.249 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:48:35.249 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:48:35.249 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T05:48:35.325 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1463: test_get_osd_id_used_by_pgs: run_mgr td/ceph-helpers x 2024-09-16T05:48:35.325 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T05:48:35.325 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T05:48:35.325 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T05:48:35.325 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T05:48:35.325 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T05:48:35.325 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T05:48:36.173 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T05:48:36.173 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:48:36.173 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:48:36.173 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:48:36.173 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:48:36.173 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:48:36.173 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:48:36.173 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:48:36.174 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:48:36.204 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1464: test_get_osd_id_used_by_pgs: run_osd td/ceph-helpers 0 2024-09-16T05:48:36.205 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T05:48:36.205 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:48:36.205 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T05:48:36.205 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:48:36.205 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T05:48:36.205 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T05:48:36.205 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:48:36.205 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:48:36.205 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:48:36.205 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T05:48:36.205 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T05:48:36.206 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:48:36.206 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:48:36.206 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T05:48:36.206 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:48:36.206 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:48:36.206 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:48:36.206 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:48:36.206 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:48:36.206 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:48:36.206 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:48:36.206 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:48:36.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:48:36.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:48:36.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:48:36.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T05:48:36.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T05:48:36.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:48:36.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:48:36.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:48:36.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:48:36.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:48:36.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:48:36.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T05:48:36.208 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:48:36.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d08b0aee-1f8d-44ab-b51c-aa6bbff3c9f2 2024-09-16T05:48:36.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d08b0aee-1f8d-44ab-b51c-aa6bbff3c9f2' 2024-09-16T05:48:36.208 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 d08b0aee-1f8d-44ab-b51c-aa6bbff3c9f2 2024-09-16T05:48:36.209 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:48:36.224 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC0xudmMiRWDRAAyDkCMdqZTu5e57BtNV04Kg== 2024-09-16T05:48:36.224 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC0xudmMiRWDRAAyDkCMdqZTu5e57BtNV04Kg=="}' 2024-09-16T05:48:36.224 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d08b0aee-1f8d-44ab-b51c-aa6bbff3c9f2 -i td/ceph-helpers/0/new.json 2024-09-16T05:48:36.484 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:48:36.494 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T05:48:36.495 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC0xudmMiRWDRAAyDkCMdqZTu5e57BtNV04Kg== --osd-uuid d08b0aee-1f8d-44ab-b51c-aa6bbff3c9f2 2024-09-16T05:48:36.523 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:48:36.523+0000 7f57e0e33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:48:36.553 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:48:36.552+0000 7f57e0e33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:48:36.554 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:48:36.554+0000 7f57e0e33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:48:36.554 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:48:36.554+0000 7f57e0e33640 -1 bdev(0x55a1b1907c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-16T05:48:36.554 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:48:36.554+0000 7f57e0e33640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T05:48:42.078 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T05:48:42.078 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:48:42.080 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T05:48:42.080 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T05:48:42.080 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:48:42.593 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:48:42.593 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T05:48:42.594 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:48:42.594 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:48:42.594 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:48:42.594 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:48:42.619 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:48:42.618+0000 7f0b381cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:48:42.621 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:48:42.620+0000 7f0b381cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:48:42.622 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:48:42.621+0000 7f0b381cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:48:43.007 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T05:48:43.008 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:48:43.008 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:48:43.008 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:48:43.008 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:48:43.008 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:48:43.008 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:48:43.009 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:48:43.009 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:48:43.009 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:48:43.382 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:48:44.215 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:48:44.215+0000 7f0b381cc640 -1 Falling back to public interface 2024-09-16T05:48:44.383 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:48:44.383 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:48:44.384 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:48:44.384 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:48:44.385 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:48:44.385 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:48:44.758 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:48:45.759 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:48:45.759 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:48:45.759 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:48:45.760 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:48:45.760 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:48:45.760 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:48:46.144 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:48:46.871 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:48:46.870+0000 7f0b381cc640 -1 osd.0 0 log_to_monitors true 2024-09-16T05:48:47.145 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:48:47.145 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:48:47.146 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:48:47.146 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:48:47.146 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:48:47.146 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:48:47.629 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:48:48.631 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:48:48.631 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:48:48.631 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:48:48.631 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:48:48.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:48:48.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:48:49.015 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:48:50.016 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:48:50.016 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:48:50.016 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:48:50.017 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:48:50.017 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:48:50.017 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:48:50.424 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:48:51.426 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:48:51.426 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:48:51.426 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:48:51.427 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:48:51.427 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:48:51.427 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:48:51.936 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:48:52.937 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:48:52.937 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:48:52.938 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:48:52.938 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:48:52.938 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:48:52.938 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:48:53.324 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:48:54.325 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:48:54.325 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:48:54.325 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:48:54.326 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:48:54.326 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:48:54.326 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:48:54.802 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:48:55.803 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:48:55.803 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:48:55.803 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:48:55.804 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:48:55.804 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:48:55.804 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:48:56.215 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:48:57.216 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:48:57.216 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:48:57.217 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:48:57.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:48:57.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:48:57.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:48:57.639 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:48:58.641 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:48:58.641 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:48:58.641 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:48:58.642 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:48:58.642 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:48:58.642 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:48:59.452 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:49:00.453 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:49:00.454 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:49:00.454 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:49:00.454 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:49:00.454 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:49:00.455 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:49:01.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:49:02.308 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:49:02.308 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:49:02.309 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:49:02.309 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:49:02.309 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:49:02.309 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:49:03.024 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:49:04.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:49:04.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:49:04.026 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:49:04.027 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:49:04.027 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:49:04.027 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:49:04.488 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:49:05.490 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:49:05.490 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:49:05.490 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:49:05.490 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:49:05.491 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:49:05.491 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:49:06.169 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:49:07.170 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:49:07.171 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:49:07.171 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:49:07.171 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:49:07.172 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:49:07.172 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:49:07.943 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:49:08.944 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:49:08.944 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:49:08.945 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:49:08.945 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:49:08.945 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:49:08.945 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:49:09.393 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:49:10.394 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:49:10.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:49:10.395 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T05:49:10.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T05:49:10.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:49:10.396 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:49:10.807 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4289039204,v1:127.0.0.1:6803/4289039204] [v2:127.0.0.1:6804/4289039204,v1:127.0.0.1:6805/4289039204] exists,up d08b0aee-1f8d-44ab-b51c-aa6bbff3c9f2 2024-09-16T05:49:10.808 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:49:10.808 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:49:10.808 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:49:10.808 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1465: test_get_osd_id_used_by_pgs: create_rbd_pool 2024-09-16T05:49:10.808 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T05:49:11.169 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' does not exist 2024-09-16T05:49:11.180 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T05:49:11.180 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T05:49:11.659 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' already exists 2024-09-16T05:49:11.671 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T05:49:12.672 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T05:49:13.686 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1466: test_get_osd_id_used_by_pgs: wait_for_clean 2024-09-16T05:49:13.686 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T05:49:13.686 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T05:49:13.686 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T05:49:13.687 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T05:49:13.687 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T05:49:13.687 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T05:49:13.687 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T05:49:13.687 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T05:49:13.687 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T05:49:13.782 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T05:49:13.782 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T05:49:13.782 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T05:49:13.782 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T05:49:13.783 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-16T05:49:13.783 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-16T05:49:14.151 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-09-16T05:49:14.152 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-16T05:49:14.152 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:49:14.152 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T05:49:14.270 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836489 2024-09-16T05:49:14.270 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836489 2024-09-16T05:49:14.270 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489' 2024-09-16T05:49:14.271 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:49:14.271 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836489 2024-09-16T05:49:14.271 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:49:14.271 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-16T05:49:14.272 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836489 2024-09-16T05:49:14.272 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:49:14.273 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.0 seq 21474836489 2024-09-16T05:49:14.273 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836489 2024-09-16T05:49:14.273 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-09-16T05:49:14.273 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:49:14.676 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836487 -lt 21474836489 2024-09-16T05:49:14.676 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-16T05:49:15.677 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T05:49:15.677 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:49:16.061 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836487 -lt 21474836489 2024-09-16T05:49:16.061 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-16T05:49:17.062 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-16T05:49:17.063 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:49:17.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836490 -lt 21474836489 2024-09-16T05:49:17.484 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T05:49:17.484 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:49:17.485 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:49:17.997 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T05:49:17.997 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T05:49:17.997 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T05:49:17.997 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T05:49:17.997 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T05:49:17.997 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T05:49:17.997 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T05:49:17.998 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T05:49:18.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T05:49:18.419 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T05:49:18.419 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:49:18.419 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:49:18.920 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T05:49:18.920 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T05:49:18.921 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T05:49:18.922 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1467: test_get_osd_id_used_by_pgs: get_osd_id_used_by_pgs 2024-09-16T05:49:18.922 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1467: test_get_osd_id_used_by_pgs: uniq 2024-09-16T05:49:18.922 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: ceph --format json pg dump pgs 2024-09-16T05:49:18.922 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: jq '.pg_stats | .[] | .up[], .acting[]' 2024-09-16T05:49:18.922 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: sort 2024-09-16T05:49:19.322 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1467: test_get_osd_id_used_by_pgs: local osd_ids=0 2024-09-16T05:49:19.322 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: test_get_osd_id_used_by_pgs: test 0 = 0 2024-09-16T05:49:19.322 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: test_get_osd_id_used_by_pgs: teardown td/ceph-helpers 2024-09-16T05:49:19.323 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:49:19.323 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:49:19.323 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:49:19.323 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:49:19.323 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:49:19.323 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:49:19.323 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:49:19.323 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:49:19.427 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:49:19.427 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:49:19.428 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:49:19.428 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:49:19.429 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:49:19.429 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:49:19.430 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:49:19.431 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:49:19.431 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:49:19.431 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:49:19.431 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:49:19.431 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:49:19.432 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:49:19.432 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:49:19.438 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:49:19.438 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:49:19.438 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:49:19.438 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:49:19.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:49:19.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:49:19.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T05:49:19.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_get_osds td/ceph-helpers 2024-09-16T05:49:19.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: test_get_osds: local dir=td/ceph-helpers 2024-09-16T05:49:19.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: test_get_osds: setup td/ceph-helpers 2024-09-16T05:49:19.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T05:49:19.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T05:49:19.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:49:19.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:49:19.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:49:19.441 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:49:19.441 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:49:19.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:49:19.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:49:19.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:49:19.442 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:49:19.442 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:49:19.442 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:49:19.442 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:49:19.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:49:19.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:49:19.443 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:49:19.444 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:49:19.444 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:49:19.445 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:49:19.445 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:49:19.445 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:49:19.446 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:49:19.446 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:49:19.447 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:49:19.447 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:49:19.447 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:49:19.447 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:49:19.448 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:49:19.448 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:49:19.448 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T05:49:19.449 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:49:19.449 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:49:19.449 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:49:19.449 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T05:49:19.451 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:49:19.452 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:49:19.452 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:49:19.452 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T05:49:19.452 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: test_get_osds: run_mon td/ceph-helpers a --osd_pool_default_size=2 2024-09-16T05:49:19.452 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T05:49:19.452 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:49:19.452 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:49:19.452 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:49:19.452 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T05:49:19.452 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2024-09-16T05:49:19.832 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:49:19.832 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:49:19.832 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:49:19.833 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:49:19.833 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:49:19.833 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:49:19.833 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:49:19.833 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-16T05:49:20.016 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:49:20.018 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:49:20.018 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:49:20.018 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:49:20.018 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:49:20.018 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:49:20.018 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:49:20.018 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:49:20.018 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:49:20.018 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:49:20.018 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:49:20.018 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:49:20.018 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:49:20.019 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:49:20.019 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T05:49:20.096 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:49:20.096 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:49:20.096 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:49:20.096 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:49:20.097 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:49:20.097 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:49:20.097 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:49:20.097 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:49:20.097 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:49:20.097 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:49:20.097 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:49:20.097 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:49:20.097 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:49:20.097 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T05:49:20.174 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: test_get_osds: run_mgr td/ceph-helpers x 2024-09-16T05:49:20.174 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T05:49:20.174 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T05:49:20.174 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T05:49:20.175 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T05:49:20.175 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T05:49:20.175 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T05:49:21.021 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T05:49:21.022 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:49:21.022 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:49:21.022 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:49:21.022 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:49:21.022 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:49:21.022 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:49:21.022 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:49:21.022 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:49:21.053 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: test_get_osds: run_osd td/ceph-helpers 0 2024-09-16T05:49:21.053 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T05:49:21.053 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:49:21.053 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T05:49:21.053 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:49:21.053 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T05:49:21.053 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T05:49:21.054 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:49:21.054 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:49:21.054 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:49:21.054 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T05:49:21.054 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T05:49:21.054 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:49:21.054 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:49:21.054 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T05:49:21.054 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:49:21.054 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:49:21.054 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:49:21.054 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:49:21.054 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:49:21.055 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:49:21.055 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:49:21.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:49:21.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:49:21.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:49:21.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:49:21.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T05:49:21.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T05:49:21.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:49:21.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:49:21.056 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:49:21.056 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:49:21.056 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:49:21.056 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:49:21.056 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T05:49:21.056 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:49:21.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4c273b82-185c-4845-8093-4de244de8d49 2024-09-16T05:49:21.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4c273b82-185c-4845-8093-4de244de8d49' 2024-09-16T05:49:21.057 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 4c273b82-185c-4845-8093-4de244de8d49 2024-09-16T05:49:21.057 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:49:21.072 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDhxudmihhLBBAA5BUII8pC/OL+obn7VdTvhg== 2024-09-16T05:49:21.073 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDhxudmihhLBBAA5BUII8pC/OL+obn7VdTvhg=="}' 2024-09-16T05:49:21.073 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4c273b82-185c-4845-8093-4de244de8d49 -i td/ceph-helpers/0/new.json 2024-09-16T05:49:21.392 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:49:21.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T05:49:21.402 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDhxudmihhLBBAA5BUII8pC/OL+obn7VdTvhg== --osd-uuid 4c273b82-185c-4845-8093-4de244de8d49 2024-09-16T05:49:21.431 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:49:21.430+0000 7fac1ea0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:49:21.460 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:49:21.460+0000 7fac1ea0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:49:21.461 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:49:21.461+0000 7fac1ea0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:49:21.462 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:49:21.462+0000 7fac1ea0d640 -1 bdev(0x56149e0d1c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-16T05:49:21.462 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:49:21.462+0000 7fac1ea0d640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T05:49:26.945 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T05:49:26.945 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:49:26.945 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T05:49:26.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T05:49:26.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:49:27.463 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:49:27.464 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T05:49:27.464 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:49:27.464 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:49:27.464 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:49:27.464 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:49:27.490 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:49:27.489+0000 7f39d040d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:49:27.515 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:49:27.514+0000 7f39d040d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:49:27.516 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:49:27.516+0000 7f39d040d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:49:27.920 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T05:49:27.920 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:49:27.920 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:49:27.921 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:49:27.921 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:49:27.921 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:49:27.921 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:49:27.921 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:49:27.921 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:49:27.921 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:49:28.329 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:49:28.862 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:49:28.862+0000 7f39d040d640 -1 Falling back to public interface 2024-09-16T05:49:29.330 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:49:29.331 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:49:29.331 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:49:29.331 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:49:29.331 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:49:29.332 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:49:29.747 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:49:30.748 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:49:30.748 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:49:30.749 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:49:30.749 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:49:30.749 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:49:30.749 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:49:31.164 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:49:32.018 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:49:32.017+0000 7f39d040d640 -1 osd.0 0 log_to_monitors true 2024-09-16T05:49:32.166 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:49:32.166 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:49:32.166 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:49:32.166 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:49:32.167 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:49:32.167 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:49:32.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:49:33.622 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:49:33.622 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:49:33.622 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:49:33.623 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:49:33.623 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:49:33.623 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:49:34.067 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:49:35.068 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:49:35.068 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:49:35.069 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:49:35.069 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:49:35.069 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:49:35.069 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:49:35.472 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:49:36.474 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:49:36.474 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:49:36.474 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:49:36.475 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:49:36.475 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:49:36.475 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:49:36.880 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:49:37.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:49:37.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:49:37.882 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:49:37.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:49:37.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:49:37.883 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:49:38.284 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:49:39.286 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:49:39.286 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:49:39.286 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:49:39.287 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:49:39.287 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:49:39.287 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:49:39.776 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:49:40.778 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:49:40.778 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:49:40.778 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:49:40.779 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:49:40.779 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:49:40.779 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:49:41.189 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:49:42.190 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:49:42.191 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:49:42.191 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:49:42.191 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:49:42.191 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:49:42.191 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:49:42.587 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:49:43.588 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:49:43.588 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:49:43.589 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:49:43.589 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:49:43.589 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:49:43.589 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:49:44.375 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:49:45.376 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:49:45.376 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:49:45.377 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:49:45.377 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:49:45.377 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:49:45.377 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:49:46.197 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:49:47.199 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:49:47.199 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:49:47.199 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:49:47.199 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:49:47.200 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:49:47.200 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:49:48.006 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:49:49.007 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:49:49.008 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:49:49.008 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:49:49.008 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:49:49.008 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:49:49.009 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:49:49.436 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:49:50.437 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:49:50.437 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:49:50.437 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:49:50.438 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:49:50.438 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:49:50.438 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:49:51.104 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:49:52.105 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:49:52.105 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:49:52.106 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:49:52.106 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:49:52.106 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:49:52.107 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:49:52.841 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:49:53.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:49:53.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:49:53.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:49:53.843 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:49:53.844 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:49:53.844 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:49:54.268 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/435540147,v1:127.0.0.1:6803/435540147] [v2:127.0.0.1:6804/435540147,v1:127.0.0.1:6805/435540147] exists,up 4c273b82-185c-4845-8093-4de244de8d49 2024-09-16T05:49:54.272 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:49:54.273 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:49:54.273 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:49:54.273 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1037: test_get_osds: run_osd td/ceph-helpers 1 2024-09-16T05:49:54.273 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T05:49:54.273 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:49:54.273 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T05:49:54.273 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:49:54.273 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-09-16T05:49:54.273 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T05:49:54.273 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:49:54.273 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:49:54.273 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:49:54.274 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-16T05:49:54.274 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-16T05:49:54.274 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:49:54.274 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:49:54.274 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T05:49:54.274 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:49:54.274 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:49:54.274 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:49:54.274 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:49:54.274 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:49:54.274 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:49:54.274 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:49:54.274 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:49:54.274 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:49:54.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:49:54.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:49:54.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T05:49:54.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T05:49:54.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:49:54.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:49:54.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:49:54.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:49:54.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:49:54.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:49:54.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-09-16T05:49:54.275 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:49:54.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=11d49892-c8bd-4948-8a44-c3468e9da3a6 2024-09-16T05:49:54.276 INFO:tasks.workunit.client.0.smithi178.stdout:add osd1 11d49892-c8bd-4948-8a44-c3468e9da3a6 2024-09-16T05:49:54.276 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 11d49892-c8bd-4948-8a44-c3468e9da3a6' 2024-09-16T05:49:54.276 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:49:54.291 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQACx+dmbeZOERAAx/t0ql27acPYJfjph2HKJw== 2024-09-16T05:49:54.291 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQACx+dmbeZOERAAx/t0ql27acPYJfjph2HKJw=="}' 2024-09-16T05:49:54.291 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 11d49892-c8bd-4948-8a44-c3468e9da3a6 -i td/ceph-helpers/1/new.json 2024-09-16T05:49:54.762 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:49:54.773 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-09-16T05:49:54.774 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQACx+dmbeZOERAAx/t0ql27acPYJfjph2HKJw== --osd-uuid 11d49892-c8bd-4948-8a44-c3468e9da3a6 2024-09-16T05:49:54.802 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:49:54.802+0000 7f70ebab6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:49:54.809 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:49:54.809+0000 7f70ebab6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:49:54.810 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:49:54.810+0000 7f70ebab6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:49:54.811 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:49:54.810+0000 7f70ebab6640 -1 bdev(0x564ac3d77c00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2024-09-16T05:49:54.811 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:49:54.810+0000 7f70ebab6640 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-16T05:50:01.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-16T05:50:01.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:50:01.031 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd1 key to auth repository 2024-09-16T05:50:01.032 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T05:50:01.032 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:50:01.554 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.1 2024-09-16T05:50:01.555 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T05:50:01.555 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:50:01.555 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:50:01.555 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:50:01.555 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:50:01.581 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:50:01.580+0000 7f79806dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:50:01.598 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:50:01.598+0000 7f79806dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:50:01.599 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:50:01.599+0000 7f79806dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:50:01.988 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T05:50:01.988 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:50:01.988 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T05:50:01.988 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:50:01.988 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:50:01.988 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:50:01.988 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:50:01.989 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:50:01.989 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:50:01.989 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:50:02.415 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:50:02.690 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:50:02.690+0000 7f79806dc640 -1 Falling back to public interface 2024-09-16T05:50:03.416 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:50:03.416 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:50:03.416 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:50:03.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:50:03.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:50:03.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:50:03.796 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:50:04.798 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:50:04.798 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:50:04.798 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:50:04.799 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:50:04.799 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:50:04.799 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:50:05.220 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:50:05.854 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:50:05.854+0000 7f79806dc640 -1 osd.1 0 log_to_monitors true 2024-09-16T05:50:06.219 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:50:06.219 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:50:06.219 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:50:06.220 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:50:06.220 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:50:06.220 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:50:06.697 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:50:07.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:50:07.699 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:50:07.699 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:50:07.699 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:50:07.699 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:50:07.699 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:50:08.203 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:50:09.204 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:50:09.204 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:50:09.205 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:50:09.205 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:50:09.205 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:50:09.205 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:50:09.753 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:50:10.755 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:50:10.755 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:50:10.755 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:50:10.756 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:50:10.756 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:50:10.756 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:50:11.140 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:50:12.141 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:50:12.141 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:50:12.142 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:50:12.142 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:50:12.142 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:50:12.142 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:50:12.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:50:13.537 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:50:13.538 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:50:13.538 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:50:13.538 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:50:13.538 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:50:13.538 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:50:13.984 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:50:14.985 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:50:14.986 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:50:14.986 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:50:14.986 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:50:14.986 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:50:14.986 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:50:15.490 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:50:16.493 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:50:16.493 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:50:16.493 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:50:16.494 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:50:16.494 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:50:16.494 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:50:17.020 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:50:18.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:50:18.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:50:18.022 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:50:18.022 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:50:18.022 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:50:18.022 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:50:18.510 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:50:19.511 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:50:19.512 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:50:19.512 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:50:19.512 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:50:19.513 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:50:19.513 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:50:20.691 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:50:21.692 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:50:21.692 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:50:21.693 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:50:21.693 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:50:21.693 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:50:21.693 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:50:22.816 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:50:23.817 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:50:23.818 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:50:23.818 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:50:23.818 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:50:23.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:50:23.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:50:24.650 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:50:25.651 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:50:25.651 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:50:25.652 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:50:25.652 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:50:25.652 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:50:25.652 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:50:26.716 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:50:27.717 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:50:27.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:50:27.718 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:50:27.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:50:27.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:50:27.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:50:28.809 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:50:29.810 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:50:29.810 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:50:29.811 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:50:29.811 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:50:29.811 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:50:29.811 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:50:30.200 INFO:tasks.workunit.client.0.smithi178.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/462164055,v1:127.0.0.1:6811/462164055] [v2:127.0.0.1:6812/462164055,v1:127.0.0.1:6813/462164055] exists,up 11d49892-c8bd-4948-8a44-c3468e9da3a6 2024-09-16T05:50:30.201 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:50:30.201 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:50:30.201 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:50:30.201 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: test_get_osds: create_rbd_pool 2024-09-16T05:50:30.201 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T05:50:30.605 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' does not exist 2024-09-16T05:50:30.616 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T05:50:30.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T05:50:31.091 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' already exists 2024-09-16T05:50:31.102 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T05:50:32.103 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T05:50:33.300 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1039: test_get_osds: wait_for_clean 2024-09-16T05:50:33.300 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T05:50:33.300 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T05:50:33.301 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T05:50:33.301 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T05:50:33.301 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T05:50:33.301 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T05:50:33.302 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T05:50:33.302 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T05:50:33.302 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T05:50:33.396 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T05:50:33.396 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T05:50:33.396 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T05:50:33.396 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T05:50:33.397 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-16T05:50:33.397 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-16T05:50:33.806 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-16T05:50:33.806 INFO:tasks.workunit.client.0.smithi178.stderr:1' 2024-09-16T05:50:33.806 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-16T05:50:33.806 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:50:33.806 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T05:50:33.921 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836496 2024-09-16T05:50:33.921 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836496 2024-09-16T05:50:33.921 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496' 2024-09-16T05:50:33.921 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:50:33.921 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T05:50:34.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672969 2024-09-16T05:50:34.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672969 2024-09-16T05:50:34.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496 1-42949672969' 2024-09-16T05:50:34.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:50:34.035 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836496 2024-09-16T05:50:34.035 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:50:34.035 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-16T05:50:34.036 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836496 2024-09-16T05:50:34.036 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:50:34.037 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836496 2024-09-16T05:50:34.037 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2024-09-16T05:50:34.037 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.0 seq 21474836496 2024-09-16T05:50:34.037 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:50:34.403 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836496 2024-09-16T05:50:34.403 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-16T05:50:35.405 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T05:50:35.405 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:50:35.774 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836496 2024-09-16T05:50:35.774 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:50:35.775 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672969 2024-09-16T05:50:35.775 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:50:35.775 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-09-16T05:50:35.776 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672969 2024-09-16T05:50:35.776 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:50:35.777 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672969 2024-09-16T05:50:35.777 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-09-16T05:50:35.777 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.1 seq 42949672969 2024-09-16T05:50:35.777 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T05:50:36.233 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672969 -lt 42949672969 2024-09-16T05:50:36.234 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T05:50:36.234 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:50:36.234 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:50:36.801 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T05:50:36.801 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T05:50:36.801 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T05:50:36.801 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T05:50:36.802 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T05:50:36.802 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T05:50:36.802 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T05:50:36.802 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T05:50:37.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T05:50:37.207 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T05:50:37.207 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:50:37.207 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:50:37.717 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T05:50:37.717 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T05:50:37.717 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T05:50:37.717 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1040: test_get_osds: create_rbd_pool 2024-09-16T05:50:37.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T05:50:38.215 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' does not exist 2024-09-16T05:50:38.226 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T05:50:38.227 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T05:50:39.202 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' already exists 2024-09-16T05:50:39.215 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T05:50:40.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T05:50:41.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1041: test_get_osds: get_osds rbd GROUP 2024-09-16T05:50:41.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=rbd 2024-09-16T05:50:41.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1041: test_get_osds: grep --quiet '^[0-1] [0-1]$' 2024-09-16T05:50:41.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=GROUP 2024-09-16T05:50:41.417 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map rbd GROUP 2024-09-16T05:50:41.417 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T05:50:41.908 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-09-16T05:50:41.908 INFO:tasks.workunit.client.0.smithi178.stderr:0' 2024-09-16T05:50:41.908 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-09-16T05:50:41.908 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1042: test_get_osds: teardown td/ceph-helpers 2024-09-16T05:50:41.908 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:50:41.908 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:50:41.908 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:50:41.908 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:50:41.908 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:50:41.908 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:50:41.909 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:50:41.909 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:50:42.014 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:50:42.014 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:50:42.015 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:50:42.015 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:50:42.016 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:50:42.016 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:50:42.017 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:50:42.017 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:50:42.017 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:50:42.018 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:50:42.018 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:50:42.018 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:50:42.019 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:50:42.019 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:50:42.029 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:50:42.030 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:50:42.030 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:50:42.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:50:42.032 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:50:42.032 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:50:42.033 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T05:50:42.033 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_get_pg td/ceph-helpers 2024-09-16T05:50:42.033 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: test_get_pg: local dir=td/ceph-helpers 2024-09-16T05:50:42.033 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: test_get_pg: setup td/ceph-helpers 2024-09-16T05:50:42.033 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T05:50:42.033 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T05:50:42.033 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:50:42.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:50:42.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:50:42.034 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:50:42.034 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:50:42.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:50:42.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:50:42.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:50:42.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:50:42.034 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:50:42.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:50:42.036 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:50:42.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:50:42.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:50:42.036 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:50:42.037 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:50:42.037 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:50:42.037 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:50:42.037 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:50:42.038 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:50:42.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:50:42.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:50:42.039 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:50:42.039 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:50:42.039 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:50:42.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:50:42.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:50:42.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:50:42.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T05:50:42.041 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:50:42.041 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:50:42.041 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:50:42.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T05:50:42.043 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:50:42.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:50:42.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:50:42.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T05:50:42.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1103: test_get_pg: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:50:42.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T05:50:42.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:50:42.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:50:42.045 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:50:42.045 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T05:50:42.045 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:50:42.403 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:50:42.404 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:50:42.404 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:50:42.404 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:50:42.404 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:50:42.404 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:50:42.404 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:50:42.404 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:50:42.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:50:42.664 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:50:42.665 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:50:42.665 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:50:42.665 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:50:42.665 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:50:42.665 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:50:42.665 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:50:42.665 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:50:42.665 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:50:42.665 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:50:42.665 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:50:42.665 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:50:42.665 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:50:42.665 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T05:50:42.744 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:50:42.744 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:50:42.744 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:50:42.744 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:50:42.744 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:50:42.744 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:50:42.744 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:50:42.744 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:50:42.744 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:50:42.744 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:50:42.745 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:50:42.745 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:50:42.745 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:50:42.745 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T05:50:42.822 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1104: test_get_pg: run_mgr td/ceph-helpers x 2024-09-16T05:50:42.822 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T05:50:42.822 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T05:50:42.822 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T05:50:42.822 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T05:50:42.822 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T05:50:42.822 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T05:50:43.667 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T05:50:43.668 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:50:43.668 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:50:43.668 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:50:43.668 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:50:43.668 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:50:43.668 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:50:43.668 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:50:43.669 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:50:43.699 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: test_get_pg: run_osd td/ceph-helpers 0 2024-09-16T05:50:43.700 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T05:50:43.700 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:50:43.700 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T05:50:43.700 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:50:43.700 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T05:50:43.700 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T05:50:43.700 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:50:43.700 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:50:43.700 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:50:43.700 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T05:50:43.701 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T05:50:43.703 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:50:43.704 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:50:43.704 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T05:50:43.704 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:50:43.704 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:50:43.704 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:50:43.704 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:50:43.704 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:50:43.704 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:50:43.704 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:50:43.704 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:50:43.704 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:50:43.704 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:50:43.705 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:50:43.705 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T05:50:43.705 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T05:50:43.705 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:50:43.705 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:50:43.705 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:50:43.705 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:50:43.705 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:50:43.705 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:50:43.705 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T05:50:43.705 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:50:43.705 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1405bde6-8c7f-47be-8d36-fc7c478bcaec 2024-09-16T05:50:43.706 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 1405bde6-8c7f-47be-8d36-fc7c478bcaec 2024-09-16T05:50:43.706 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1405bde6-8c7f-47be-8d36-fc7c478bcaec' 2024-09-16T05:50:43.706 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:50:43.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAzx+dmVBrPKhAABnmNvfwTXHsCAfQ7WXrktA== 2024-09-16T05:50:43.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAzx+dmVBrPKhAABnmNvfwTXHsCAfQ7WXrktA=="}' 2024-09-16T05:50:43.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1405bde6-8c7f-47be-8d36-fc7c478bcaec -i td/ceph-helpers/0/new.json 2024-09-16T05:50:43.995 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:50:44.005 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T05:50:44.006 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAzx+dmVBrPKhAABnmNvfwTXHsCAfQ7WXrktA== --osd-uuid 1405bde6-8c7f-47be-8d36-fc7c478bcaec 2024-09-16T05:50:44.035 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:50:44.034+0000 7f387f60e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:50:44.089 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:50:44.089+0000 7f387f60e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:50:44.090 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:50:44.090+0000 7f387f60e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:50:44.091 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:50:44.091+0000 7f387f60e640 -1 bdev(0x560f92964800 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-16T05:50:44.091 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:50:44.091+0000 7f387f60e640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T05:50:49.827 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T05:50:49.827 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:50:49.828 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T05:50:49.828 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T05:50:49.828 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:50:50.337 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:50:50.338 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T05:50:50.338 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:50:50.338 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:50:50.338 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:50:50.338 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:50:50.366 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:50:50.365+0000 7f2798a1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:50:50.384 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:50:50.383+0000 7f2798a1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:50:50.384 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:50:50.384+0000 7f2798a1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:50:50.788 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T05:50:50.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:50:50.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:50:50.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:50:50.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:50:50.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:50:50.789 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:50:50.790 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:50:50.790 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:50:50.790 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:50:51.206 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:50:51.474 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:50:51.474+0000 7f2798a1c640 -1 Falling back to public interface 2024-09-16T05:50:52.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:50:52.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:50:52.208 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:50:52.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:50:52.209 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:50:52.209 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:50:52.631 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:50:53.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:50:53.633 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:50:53.633 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:50:53.633 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:50:53.634 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:50:53.634 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:50:54.017 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:50:54.276 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:50:54.275+0000 7f2798a1c640 -1 osd.0 0 log_to_monitors true 2024-09-16T05:50:55.018 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:50:55.018 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:50:55.019 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:50:55.019 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:50:55.019 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:50:55.020 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:50:55.471 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:50:56.473 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:50:56.473 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:50:56.473 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:50:56.474 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:50:56.474 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:50:56.474 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:50:56.851 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:50:57.853 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:50:57.853 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:50:57.853 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:50:57.854 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:50:57.854 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:50:57.854 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:50:58.365 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:50:59.366 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:50:59.366 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:50:59.367 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:50:59.367 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:50:59.367 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:50:59.367 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:50:59.752 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:51:00.753 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:51:00.753 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:51:00.754 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:51:00.754 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:51:00.754 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:51:00.754 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:51:01.289 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:51:02.290 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:51:02.290 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:51:02.291 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:51:02.292 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:51:02.292 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:51:02.292 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:51:02.697 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:51:03.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:51:03.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:51:03.699 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:51:03.699 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:51:03.699 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:51:03.699 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:51:04.112 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:51:05.114 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:51:05.114 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:51:05.114 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:51:05.114 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:51:05.115 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:51:05.115 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:51:05.495 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:51:06.496 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:51:06.496 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:51:06.496 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:51:06.497 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:51:06.497 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:51:06.497 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:51:07.123 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:51:08.124 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:51:08.124 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:51:08.125 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:51:08.125 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:51:08.125 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:51:08.125 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:51:08.977 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:51:09.978 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:51:09.978 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:51:09.978 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:51:09.979 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:51:09.979 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:51:09.979 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:51:10.935 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:51:11.936 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:51:11.936 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:51:11.937 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:51:11.938 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:51:11.938 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:51:11.938 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:51:12.695 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:51:13.696 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:51:13.697 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:51:13.697 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:51:13.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:51:13.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:51:13.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:51:14.505 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:51:15.506 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:51:15.506 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:51:15.506 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:51:15.507 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:51:15.507 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:51:15.507 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:51:16.285 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:51:17.286 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:51:17.287 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:51:17.287 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:51:17.287 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:51:17.288 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:51:17.288 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:51:17.711 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4236622494,v1:127.0.0.1:6803/4236622494] [v2:127.0.0.1:6804/4236622494,v1:127.0.0.1:6805/4236622494] exists,up 1405bde6-8c7f-47be-8d36-fc7c478bcaec 2024-09-16T05:51:17.712 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:51:17.712 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:51:17.713 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:51:17.713 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: test_get_pg: create_rbd_pool 2024-09-16T05:51:17.713 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T05:51:18.152 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' does not exist 2024-09-16T05:51:18.163 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T05:51:18.164 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T05:51:18.629 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' already exists 2024-09-16T05:51:18.641 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T05:51:19.642 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T05:51:20.774 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: test_get_pg: wait_for_clean 2024-09-16T05:51:20.774 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T05:51:20.775 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T05:51:20.775 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T05:51:20.775 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T05:51:20.775 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T05:51:20.776 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T05:51:20.776 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T05:51:20.776 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T05:51:20.776 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T05:51:20.870 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T05:51:20.871 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T05:51:20.871 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T05:51:20.871 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T05:51:20.871 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-16T05:51:20.871 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-16T05:51:21.238 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-09-16T05:51:21.238 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-16T05:51:21.238 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:51:21.238 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T05:51:21.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836486 2024-09-16T05:51:21.354 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836486 2024-09-16T05:51:21.354 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836486' 2024-09-16T05:51:21.354 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:51:21.354 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836486 2024-09-16T05:51:21.354 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:51:21.355 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-16T05:51:21.355 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836486 2024-09-16T05:51:21.355 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:51:21.356 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836486 2024-09-16T05:51:21.356 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.0 seq 21474836486 2024-09-16T05:51:21.356 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-09-16T05:51:21.356 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:51:21.757 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836486 -lt 21474836486 2024-09-16T05:51:21.757 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T05:51:21.757 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:51:21.758 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:51:22.298 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T05:51:22.298 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T05:51:22.298 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T05:51:22.298 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T05:51:22.298 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T05:51:22.298 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T05:51:22.299 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T05:51:22.299 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T05:51:22.704 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T05:51:22.704 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T05:51:22.704 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:51:22.705 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:51:23.292 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T05:51:23.292 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T05:51:23.292 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T05:51:23.292 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1108: test_get_pg: get_pg rbd GROUP 2024-09-16T05:51:23.292 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-09-16T05:51:23.293 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1108: test_get_pg: grep --quiet '^[0-9]\.[0-9a-f][0-9a-f]*$' 2024-09-16T05:51:23.293 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=GROUP 2024-09-16T05:51:23.293 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd GROUP 2024-09-16T05:51:23.293 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-16T05:51:23.727 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1109: test_get_pg: teardown td/ceph-helpers 2024-09-16T05:51:23.727 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:51:23.727 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:51:23.727 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:51:23.727 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:51:23.727 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:51:23.727 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:51:23.727 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:51:23.727 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:51:23.832 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:51:23.833 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:51:23.833 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:51:23.834 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:51:23.835 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:51:23.835 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:51:23.835 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:51:23.836 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:51:23.836 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:51:23.836 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:51:23.836 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:51:23.837 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:51:23.838 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:51:23.838 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:51:23.848 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:51:23.848 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:51:23.849 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:51:23.849 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:51:23.849 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:51:23.849 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:51:23.849 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T05:51:23.849 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_get_primary td/ceph-helpers 2024-09-16T05:51:23.849 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1204: test_get_primary: local dir=td/ceph-helpers 2024-09-16T05:51:23.850 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: test_get_primary: setup td/ceph-helpers 2024-09-16T05:51:23.850 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T05:51:23.850 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T05:51:23.850 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:51:23.850 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:51:23.850 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:51:23.850 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:51:23.850 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:51:23.851 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:51:23.851 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:51:23.851 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:51:23.851 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:51:23.851 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:51:23.851 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:51:23.851 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:51:23.856 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:51:23.856 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:51:23.856 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:51:23.856 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:51:23.856 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:51:23.856 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:51:23.857 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:51:23.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:51:23.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:51:23.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:51:23.857 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:51:23.857 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:51:23.857 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:51:23.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:51:23.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:51:23.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:51:23.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T05:51:23.858 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:51:23.858 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:51:23.858 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:51:23.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T05:51:23.860 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:51:23.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:51:23.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:51:23.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T05:51:23.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: test_get_primary: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:51:23.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T05:51:23.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:51:23.862 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:51:23.862 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:51:23.862 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T05:51:23.862 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:51:24.254 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:51:24.255 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:51:24.255 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:51:24.255 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:51:24.255 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:51:24.255 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:51:24.255 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:51:24.255 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:51:24.446 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:51:24.448 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:51:24.448 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:51:24.449 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:51:24.449 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:51:24.449 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:51:24.449 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:51:24.449 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:51:24.449 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:51:24.449 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:51:24.449 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:51:24.450 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:51:24.450 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:51:24.450 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:51:24.450 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T05:51:24.527 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:51:24.527 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:51:24.527 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:51:24.527 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:51:24.527 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:51:24.527 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:51:24.528 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:51:24.528 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:51:24.528 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:51:24.528 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:51:24.528 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:51:24.529 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:51:24.529 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:51:24.529 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T05:51:24.606 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: test_get_primary: local osd=0 2024-09-16T05:51:24.606 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: test_get_primary: run_mgr td/ceph-helpers x 2024-09-16T05:51:24.606 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T05:51:24.606 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T05:51:24.606 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T05:51:24.607 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T05:51:24.607 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T05:51:24.607 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T05:51:25.450 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T05:51:25.451 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:51:25.451 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:51:25.451 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:51:25.451 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:51:25.451 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:51:25.451 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:51:25.451 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:51:25.451 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:51:25.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: test_get_primary: run_osd td/ceph-helpers 0 2024-09-16T05:51:25.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T05:51:25.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:51:25.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T05:51:25.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:51:25.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T05:51:25.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T05:51:25.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:51:25.483 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:51:25.483 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:51:25.483 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T05:51:25.483 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T05:51:25.483 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:51:25.483 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:51:25.483 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T05:51:25.483 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:51:25.483 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:51:25.483 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:51:25.483 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:51:25.483 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:51:25.483 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:51:25.483 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:51:25.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:51:25.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:51:25.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:51:25.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:51:25.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T05:51:25.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T05:51:25.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:51:25.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:51:25.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:51:25.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:51:25.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:51:25.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:51:25.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T05:51:25.484 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:51:25.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=095d1a7a-1f86-4a47-84f3-2a0ea52221eb 2024-09-16T05:51:25.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 095d1a7a-1f86-4a47-84f3-2a0ea52221eb' 2024-09-16T05:51:25.485 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 095d1a7a-1f86-4a47-84f3-2a0ea52221eb 2024-09-16T05:51:25.486 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:51:25.502 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBdx+dmNhngHRAAUQHUB8h5gcIvs7O7bDlbdg== 2024-09-16T05:51:25.502 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBdx+dmNhngHRAAUQHUB8h5gcIvs7O7bDlbdg=="}' 2024-09-16T05:51:25.502 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 095d1a7a-1f86-4a47-84f3-2a0ea52221eb -i td/ceph-helpers/0/new.json 2024-09-16T05:51:25.796 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:51:25.805 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T05:51:25.806 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBdx+dmNhngHRAAUQHUB8h5gcIvs7O7bDlbdg== --osd-uuid 095d1a7a-1f86-4a47-84f3-2a0ea52221eb 2024-09-16T05:51:25.835 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:51:25.835+0000 7f2a6de15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:51:25.865 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:51:25.864+0000 7f2a6de15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:51:25.866 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:51:25.865+0000 7f2a6de15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:51:25.866 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:51:25.866+0000 7f2a6de15640 -1 bdev(0x556cc57b7c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-16T05:51:25.866 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:51:25.866+0000 7f2a6de15640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T05:51:31.168 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T05:51:31.168 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:51:31.169 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T05:51:31.169 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T05:51:31.169 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:51:31.697 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:51:31.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T05:51:31.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:51:31.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:51:31.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:51:31.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:51:31.724 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:51:31.724+0000 7f2d13345640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:51:31.737 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:51:31.736+0000 7f2d13345640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:51:31.738 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:51:31.737+0000 7f2d13345640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:51:32.072 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T05:51:32.072 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:51:32.072 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:51:32.072 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:51:32.072 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:51:32.072 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:51:32.073 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:51:32.073 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:51:32.073 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:51:32.073 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:51:32.446 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:51:33.330 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:51:33.329+0000 7f2d13345640 -1 Falling back to public interface 2024-09-16T05:51:33.448 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:51:33.448 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:51:33.448 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:51:33.448 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:51:33.449 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:51:33.449 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:51:33.876 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:51:34.877 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:51:34.877 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:51:34.877 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:51:34.878 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:51:34.878 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:51:34.878 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:51:35.294 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:51:35.998 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:51:35.997+0000 7f2d13345640 -1 osd.0 0 log_to_monitors true 2024-09-16T05:51:36.295 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:51:36.295 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:51:36.296 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:51:36.296 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:51:36.296 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:51:36.296 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:51:36.724 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:51:37.725 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:51:37.725 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:51:37.725 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:51:37.726 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:51:37.726 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:51:37.726 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:51:38.129 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:51:39.131 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:51:39.131 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:51:39.131 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:51:39.132 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:51:39.132 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:51:39.132 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:51:39.552 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:51:40.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:51:40.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:51:40.553 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:51:40.554 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:51:40.554 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:51:40.554 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:51:41.058 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:51:42.059 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:51:42.059 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:51:42.060 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:51:42.060 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:51:42.060 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:51:42.060 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:51:42.456 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:51:43.457 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:51:43.458 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:51:43.458 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:51:43.458 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:51:43.458 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:51:43.459 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:51:43.948 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:51:44.949 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:51:44.949 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:51:44.950 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:51:44.950 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:51:44.950 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:51:44.950 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:51:45.368 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:51:46.369 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:51:46.369 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:51:46.369 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:51:46.370 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:51:46.370 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:51:46.370 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:51:46.791 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:51:47.792 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:51:47.793 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:51:47.793 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:51:47.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:51:47.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:51:47.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:51:48.295 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:51:49.297 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:51:49.297 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:51:49.297 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:51:49.298 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:51:49.298 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:51:49.298 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:51:50.006 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:51:51.008 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:51:51.008 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:51:51.008 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:51:51.009 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:51:51.009 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:51:51.009 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:51:51.641 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:51:52.642 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:51:52.643 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:51:52.643 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:51:52.643 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:51:52.643 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:51:52.644 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:51:53.394 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:51:54.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:51:54.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:51:54.395 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:51:54.396 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:51:54.396 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:51:54.396 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:51:55.163 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:51:56.165 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:51:56.165 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:51:56.165 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:51:56.166 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:51:56.166 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:51:56.166 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:51:57.260 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:51:58.261 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:51:58.261 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:51:58.262 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:51:58.262 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:51:58.262 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:51:58.262 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:51:58.722 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:51:59.723 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:51:59.723 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:51:59.724 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T05:51:59.724 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T05:51:59.724 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:51:59.724 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:52:00.097 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1381625465,v1:127.0.0.1:6803/1381625465] [v2:127.0.0.1:6804/1381625465,v1:127.0.0.1:6805/1381625465] exists,up 095d1a7a-1f86-4a47-84f3-2a0ea52221eb 2024-09-16T05:52:00.098 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:52:00.098 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:52:00.098 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:52:00.098 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: test_get_primary: create_rbd_pool 2024-09-16T05:52:00.098 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T05:52:00.499 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' does not exist 2024-09-16T05:52:00.510 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T05:52:00.510 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T05:52:00.976 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' already exists 2024-09-16T05:52:00.987 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T05:52:01.989 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T05:52:03.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: test_get_primary: wait_for_clean 2024-09-16T05:52:03.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T05:52:03.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T05:52:03.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T05:52:03.027 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T05:52:03.027 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T05:52:03.027 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T05:52:03.028 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T05:52:03.028 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T05:52:03.028 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T05:52:03.123 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T05:52:03.123 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T05:52:03.123 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T05:52:03.123 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T05:52:03.123 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-16T05:52:03.123 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-16T05:52:03.495 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-09-16T05:52:03.495 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-16T05:52:03.495 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:52:03.495 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T05:52:03.609 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836488 2024-09-16T05:52:03.610 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836488 2024-09-16T05:52:03.610 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836488' 2024-09-16T05:52:03.610 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:52:03.610 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836488 2024-09-16T05:52:03.610 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:52:03.610 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-16T05:52:03.611 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836488 2024-09-16T05:52:03.611 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:52:03.612 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836488 2024-09-16T05:52:03.612 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.0 seq 21474836488 2024-09-16T05:52:03.612 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2024-09-16T05:52:03.612 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:52:03.995 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836486 -lt 21474836488 2024-09-16T05:52:03.995 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-16T05:52:04.996 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T05:52:04.997 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:52:05.439 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836486 -lt 21474836488 2024-09-16T05:52:05.439 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-16T05:52:06.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-16T05:52:06.440 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:52:06.822 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836489 -lt 21474836488 2024-09-16T05:52:06.822 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T05:52:06.823 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:52:06.823 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:52:07.327 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T05:52:07.327 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T05:52:07.327 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T05:52:07.327 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T05:52:07.328 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T05:52:07.328 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T05:52:07.328 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T05:52:07.328 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T05:52:07.750 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T05:52:07.750 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T05:52:07.750 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:52:07.751 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:52:08.250 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T05:52:08.250 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T05:52:08.250 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T05:52:08.250 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: test_get_primary: get_primary rbd GROUP 2024-09-16T05:52:08.250 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-09-16T05:52:08.250 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=GROUP 2024-09-16T05:52:08.250 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd GROUP 2024-09-16T05:52:08.250 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-16T05:52:08.637 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: test_get_primary: test 0 = 0 2024-09-16T05:52:08.638 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: test_get_primary: teardown td/ceph-helpers 2024-09-16T05:52:08.638 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:52:08.638 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:52:08.638 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:52:08.638 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:52:08.638 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:52:08.638 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:52:08.638 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:52:08.638 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:52:08.743 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:52:08.743 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:52:08.744 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:52:08.744 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:52:08.745 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:52:08.745 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:52:08.746 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:52:08.746 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:52:08.746 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:52:08.747 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:52:08.747 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:52:08.747 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:52:08.748 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:52:08.749 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:52:08.755 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:52:08.755 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:52:08.755 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:52:08.755 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:52:08.756 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:52:08.756 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:52:08.756 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T05:52:08.756 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_get_timeout_delays td/ceph-helpers 2024-09-16T05:52:08.756 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1616: test_get_timeout_delays: get_timeout_delays 1 2024-09-16T05:52:08.757 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T05:52:08.757 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T05:52:08.757 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T05:52:08.757 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T05:52:08.757 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T05:52:08.766 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1616: test_get_timeout_delays: test '1 ' = '1 ' 2024-09-16T05:52:08.767 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1617: test_get_timeout_delays: get_timeout_delays 5 2024-09-16T05:52:08.768 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T05:52:08.768 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T05:52:08.768 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T05:52:08.768 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T05:52:08.768 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T05:52:08.786 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1617: test_get_timeout_delays: test '1 2 2 ' = '1 2 2 ' 2024-09-16T05:52:08.787 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1618: test_get_timeout_delays: get_timeout_delays 6 2024-09-16T05:52:08.787 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T05:52:08.787 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T05:52:08.787 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T05:52:08.787 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T05:52:08.787 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T05:52:08.805 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1618: test_get_timeout_delays: test '1 2 3 ' = '1 2 3 ' 2024-09-16T05:52:08.805 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1619: test_get_timeout_delays: get_timeout_delays 7 2024-09-16T05:52:08.806 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T05:52:08.806 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T05:52:08.806 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T05:52:08.806 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T05:52:08.806 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T05:52:08.829 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1619: test_get_timeout_delays: test '1 2 4 ' = '1 2 4 ' 2024-09-16T05:52:08.830 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1620: test_get_timeout_delays: get_timeout_delays 8 2024-09-16T05:52:08.830 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T05:52:08.830 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T05:52:08.830 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T05:52:08.830 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T05:52:08.830 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T05:52:08.855 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1620: test_get_timeout_delays: test '1 2 4 1 ' = '1 2 4 1 ' 2024-09-16T05:52:08.855 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1621: test_get_timeout_delays: get_timeout_delays 1 .1 2024-09-16T05:52:08.856 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T05:52:08.856 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T05:52:08.856 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T05:52:08.856 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T05:52:08.856 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T05:52:08.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1621: test_get_timeout_delays: test '0.1 0.2 0.4 0.3 ' = '0.1 0.2 0.4 0.3 ' 2024-09-16T05:52:08.881 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1622: test_get_timeout_delays: get_timeout_delays 1.5 .1 2024-09-16T05:52:08.882 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T05:52:08.882 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T05:52:08.882 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T05:52:08.882 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T05:52:08.882 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T05:52:08.912 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1622: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 ' = '0.1 0.2 0.4 0.8 ' 2024-09-16T05:52:08.913 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: test_get_timeout_delays: get_timeout_delays 5 .1 2024-09-16T05:52:08.913 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T05:52:08.913 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T05:52:08.913 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T05:52:08.913 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T05:52:08.914 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T05:52:08.952 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 1.9 ' = '0.1 0.2 0.4 0.8 1.6 1.9 ' 2024-09-16T05:52:08.952 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1624: test_get_timeout_delays: get_timeout_delays 6 .1 2024-09-16T05:52:08.952 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T05:52:08.953 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T05:52:08.953 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T05:52:08.953 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T05:52:08.953 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T05:52:08.991 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1624: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 2.9 ' = '0.1 0.2 0.4 0.8 1.6 2.9 ' 2024-09-16T05:52:08.991 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1625: test_get_timeout_delays: get_timeout_delays 6.3 .1 2024-09-16T05:52:08.991 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T05:52:08.991 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T05:52:08.992 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T05:52:08.992 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T05:52:08.992 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T05:52:09.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1625: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 ' = '0.1 0.2 0.4 0.8 1.6 3.2 ' 2024-09-16T05:52:09.031 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1626: test_get_timeout_delays: get_timeout_delays 20 .1 2024-09-16T05:52:09.031 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T05:52:09.031 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T05:52:09.031 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T05:52:09.031 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T05:52:09.031 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T05:52:09.081 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1626: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 7.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 7.3 ' 2024-09-16T05:52:09.082 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: test_get_timeout_delays: get_timeout_delays 300 .1 0 2024-09-16T05:52:09.082 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T05:52:09.083 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T05:52:09.083 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T05:52:09.083 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T05:52:09.083 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T05:52:09.161 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 12.8 25.6 51.2 102.4 95.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 12.8 25.6 51.2 102.4 95.3 ' 2024-09-16T05:52:09.161 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1628: test_get_timeout_delays: get_timeout_delays 300 .1 10 2024-09-16T05:52:09.162 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T05:52:09.162 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T05:52:09.162 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T05:52:09.162 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T05:52:09.162 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T05:52:09.398 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1628: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 7.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 7.3 ' 2024-09-16T05:52:09.399 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T05:52:09.399 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_is_clean td/ceph-helpers 2024-09-16T05:52:09.399 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1561: test_is_clean: local dir=td/ceph-helpers 2024-09-16T05:52:09.399 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: test_is_clean: setup td/ceph-helpers 2024-09-16T05:52:09.399 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T05:52:09.399 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T05:52:09.399 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:52:09.399 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:52:09.399 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:52:09.399 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:52:09.399 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:52:09.399 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:52:09.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:52:09.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:52:09.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:52:09.400 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:52:09.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:52:09.401 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:52:09.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:52:09.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:52:09.401 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:52:09.402 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:52:09.403 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:52:09.403 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:52:09.403 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:52:09.403 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:52:09.404 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:52:09.404 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:52:09.406 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:52:09.406 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:52:09.406 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:52:09.406 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:52:09.407 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:52:09.407 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:52:09.407 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T05:52:09.408 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:52:09.408 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:52:09.408 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:52:09.408 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T05:52:09.410 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:52:09.410 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:52:09.410 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:52:09.410 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T05:52:09.410 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: test_is_clean: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:52:09.411 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T05:52:09.411 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:52:09.411 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:52:09.411 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:52:09.411 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T05:52:09.411 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:52:09.814 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:52:09.814 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:52:09.814 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:52:09.814 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:52:09.814 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:52:09.814 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:52:09.815 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:52:09.815 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:52:10.006 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:52:10.008 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:52:10.008 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:52:10.009 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:52:10.009 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:52:10.009 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:52:10.009 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:52:10.009 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:52:10.009 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:52:10.009 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:52:10.009 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:52:10.009 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:52:10.009 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:52:10.009 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:52:10.009 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T05:52:10.088 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:52:10.089 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:52:10.089 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:52:10.089 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:52:10.089 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:52:10.089 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:52:10.089 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:52:10.089 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:52:10.089 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:52:10.089 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:52:10.089 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:52:10.089 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:52:10.089 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:52:10.089 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T05:52:10.170 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1565: test_is_clean: run_mgr td/ceph-helpers x 2024-09-16T05:52:10.170 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T05:52:10.170 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T05:52:10.170 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T05:52:10.170 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T05:52:10.170 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T05:52:10.171 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T05:52:11.010 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T05:52:11.010 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:52:11.010 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:52:11.010 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:52:11.010 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:52:11.010 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:52:11.011 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:52:11.011 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:52:11.011 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:52:11.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: test_is_clean: run_osd td/ceph-helpers 0 2024-09-16T05:52:11.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T05:52:11.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:52:11.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T05:52:11.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:52:11.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T05:52:11.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T05:52:11.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:52:11.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:52:11.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:52:11.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T05:52:11.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T05:52:11.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:52:11.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:52:11.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T05:52:11.043 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:52:11.043 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:52:11.043 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:52:11.043 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:52:11.043 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:52:11.043 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:52:11.043 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:52:11.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:52:11.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:52:11.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:52:11.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:52:11.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T05:52:11.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T05:52:11.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:52:11.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:52:11.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:52:11.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:52:11.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:52:11.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:52:11.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T05:52:11.044 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:52:11.045 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ad1b83d6-12bf-4a90-b321-3899b3bf858f 2024-09-16T05:52:11.045 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 ad1b83d6-12bf-4a90-b321-3899b3bf858f 2024-09-16T05:52:11.045 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ad1b83d6-12bf-4a90-b321-3899b3bf858f' 2024-09-16T05:52:11.045 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:52:11.061 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCLx+dmZfacAxAAk1/t0c40jpk693FJvvM70g== 2024-09-16T05:52:11.061 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCLx+dmZfacAxAAk1/t0c40jpk693FJvvM70g=="}' 2024-09-16T05:52:11.061 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ad1b83d6-12bf-4a90-b321-3899b3bf858f -i td/ceph-helpers/0/new.json 2024-09-16T05:52:11.339 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:52:11.349 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T05:52:11.350 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCLx+dmZfacAxAAk1/t0c40jpk693FJvvM70g== --osd-uuid ad1b83d6-12bf-4a90-b321-3899b3bf858f 2024-09-16T05:52:11.379 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:52:11.377+0000 7f00c6ef1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:52:11.408 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:52:11.407+0000 7f00c6ef1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:52:11.409 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:52:11.408+0000 7f00c6ef1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:52:11.410 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:52:11.408+0000 7f00c6ef1640 -1 bdev(0x555c49655c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-16T05:52:11.410 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:52:11.409+0000 7f00c6ef1640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T05:52:17.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T05:52:17.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:52:17.441 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T05:52:17.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T05:52:17.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:52:17.956 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T05:52:17.957 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:52:17.957 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:52:17.957 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:52:17.957 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:52:17.957 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:52:17.983 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:52:17.983+0000 7f9fd0cd6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:52:18.010 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:52:18.010+0000 7f9fd0cd6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:52:18.011 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:52:18.011+0000 7f9fd0cd6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:52:18.374 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T05:52:18.374 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:52:18.374 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:52:18.375 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:52:18.375 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:52:18.375 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:52:18.375 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:52:18.375 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:52:18.375 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:52:18.375 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:52:18.744 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:52:19.595 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:52:19.595+0000 7f9fd0cd6640 -1 Falling back to public interface 2024-09-16T05:52:19.745 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:52:19.745 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:52:19.746 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:52:19.746 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:52:19.746 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:52:19.746 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:52:20.129 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:52:21.130 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:52:21.130 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:52:21.130 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:52:21.131 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:52:21.131 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:52:21.131 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:52:21.550 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:52:22.321 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:52:22.320+0000 7f9fd0cd6640 -1 osd.0 0 log_to_monitors true 2024-09-16T05:52:22.552 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:52:22.552 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:52:22.552 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:52:22.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:52:22.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:52:22.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:52:23.014 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:52:24.015 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:52:24.015 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:52:24.016 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:52:24.017 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:52:24.017 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:52:24.017 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:52:24.422 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:52:25.423 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:52:25.424 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:52:25.424 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:52:25.424 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:52:25.424 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:52:25.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:52:25.810 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:52:26.811 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:52:26.812 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:52:26.812 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:52:26.812 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:52:26.813 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:52:26.813 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:52:27.320 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:52:28.321 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:52:28.322 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:52:28.322 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:52:28.322 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:52:28.322 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:52:28.323 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:52:28.708 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:52:29.710 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:52:29.710 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:52:29.710 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:52:29.711 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:52:29.711 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:52:29.711 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:52:30.152 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:52:31.153 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:52:31.153 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:52:31.154 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:52:31.154 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:52:31.154 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:52:31.154 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:52:31.561 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:52:32.562 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:52:32.562 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:52:32.563 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:52:32.564 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:52:32.564 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:52:32.564 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:52:33.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:52:34.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:52:34.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:52:34.045 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:52:34.045 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:52:34.045 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:52:34.045 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:52:34.904 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:52:35.905 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:52:35.905 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:52:35.906 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:52:35.906 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:52:35.906 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:52:35.906 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:52:36.700 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:52:37.702 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:52:37.702 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:52:37.702 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:52:37.702 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:52:37.703 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:52:37.703 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:52:38.386 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:52:39.388 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:52:39.388 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:52:39.388 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:52:39.389 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:52:39.389 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:52:39.389 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:52:40.152 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:52:41.153 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:52:41.153 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:52:41.154 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:52:41.154 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:52:41.154 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:52:41.155 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:52:42.299 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:52:43.300 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:52:43.301 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:52:43.301 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:52:43.301 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:52:43.302 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:52:43.302 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:52:44.216 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:52:45.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:52:45.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:52:45.218 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:52:45.218 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:52:45.218 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:52:45.218 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:52:45.620 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2193909328,v1:127.0.0.1:6803/2193909328] [v2:127.0.0.1:6804/2193909328,v1:127.0.0.1:6805/2193909328] exists,up ad1b83d6-12bf-4a90-b321-3899b3bf858f 2024-09-16T05:52:45.620 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:52:45.620 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:52:45.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:52:45.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1567: test_is_clean: create_rbd_pool 2024-09-16T05:52:45.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T05:52:45.982 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' does not exist 2024-09-16T05:52:45.993 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T05:52:45.993 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T05:52:46.455 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' already exists 2024-09-16T05:52:46.466 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T05:52:47.468 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T05:52:48.517 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1568: test_is_clean: wait_for_clean 2024-09-16T05:52:48.517 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T05:52:48.517 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T05:52:48.517 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T05:52:48.518 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T05:52:48.518 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T05:52:48.518 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T05:52:48.518 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T05:52:48.518 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T05:52:48.518 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T05:52:48.612 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T05:52:48.612 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T05:52:48.612 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T05:52:48.612 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T05:52:48.613 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-16T05:52:48.613 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-16T05:52:48.998 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-09-16T05:52:48.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-16T05:52:48.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:52:48.999 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T05:52:49.113 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836485 2024-09-16T05:52:49.114 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836485 2024-09-16T05:52:49.114 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836485' 2024-09-16T05:52:49.114 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:52:49.114 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836485 2024-09-16T05:52:49.114 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:52:49.114 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-16T05:52:49.115 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836485 2024-09-16T05:52:49.115 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:52:49.116 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836485 2024-09-16T05:52:49.116 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2024-09-16T05:52:49.116 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.0 seq 21474836485 2024-09-16T05:52:49.117 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:52:49.495 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836483 -lt 21474836485 2024-09-16T05:52:49.495 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-16T05:52:50.496 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T05:52:50.496 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:52:50.918 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836483 -lt 21474836485 2024-09-16T05:52:50.919 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-16T05:52:51.920 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-16T05:52:51.920 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:52:52.345 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836486 -lt 21474836485 2024-09-16T05:52:52.346 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T05:52:52.346 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:52:52.346 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:52:52.848 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T05:52:52.849 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T05:52:52.849 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T05:52:52.849 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T05:52:52.849 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T05:52:52.849 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T05:52:52.849 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T05:52:52.849 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T05:52:53.228 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T05:52:53.228 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T05:52:53.228 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:52:53.229 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:52:53.726 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T05:52:53.726 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T05:52:53.726 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T05:52:53.726 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1569: test_is_clean: is_clean 2024-09-16T05:52:53.726 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: is_clean: get_num_pgs 2024-09-16T05:52:53.727 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:52:53.727 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:52:54.260 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: is_clean: num_pgs=4 2024-09-16T05:52:54.260 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1556: is_clean: test 4 '!=' 0 2024-09-16T05:52:54.260 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1557: is_clean: get_num_active_clean 2024-09-16T05:52:54.260 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T05:52:54.260 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T05:52:54.260 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T05:52:54.260 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T05:52:54.260 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T05:52:54.629 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1557: is_clean: test 4 = 4 2024-09-16T05:52:54.630 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1570: test_is_clean: teardown td/ceph-helpers 2024-09-16T05:52:54.630 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:52:54.630 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:52:54.630 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:52:54.630 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:52:54.630 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:52:54.630 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:52:54.630 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:52:54.630 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:52:54.735 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:52:54.735 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:52:54.736 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:52:54.736 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:52:54.737 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:52:54.737 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:52:54.738 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:52:54.738 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:52:54.738 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:52:54.739 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:52:54.739 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:52:54.739 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:52:54.740 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:52:54.741 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:52:54.746 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:52:54.746 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:52:54.747 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:52:54.747 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:52:54.749 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:52:54.749 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:52:54.749 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T05:52:54.749 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_kill_daemon td/ceph-helpers 2024-09-16T05:52:54.749 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:275: test_kill_daemon: local dir=td/ceph-helpers 2024-09-16T05:52:54.749 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:276: test_kill_daemon: setup td/ceph-helpers 2024-09-16T05:52:54.749 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T05:52:54.749 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T05:52:54.749 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:52:54.749 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:52:54.749 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:52:54.749 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:52:54.749 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:52:54.750 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:52:54.750 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:52:54.750 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:52:54.750 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:52:54.750 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:52:54.751 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:52:54.751 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:52:54.752 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:52:54.752 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:52:54.752 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:52:54.753 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:52:54.753 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:52:54.753 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:52:54.753 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:52:54.754 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:52:54.755 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:52:54.755 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:52:54.756 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:52:54.756 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:52:54.756 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:52:54.756 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:52:54.757 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:52:54.757 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:52:54.757 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T05:52:54.758 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:52:54.758 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:52:54.758 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:52:54.758 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T05:52:54.760 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:52:54.760 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:52:54.760 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:52:54.761 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T05:52:54.761 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:277: test_kill_daemon: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:52:54.761 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T05:52:54.761 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:52:54.761 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:52:54.761 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:52:54.761 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T05:52:54.761 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:52:55.136 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:52:55.136 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:52:55.136 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:52:55.137 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:52:55.137 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:52:55.137 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:52:55.137 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:52:55.137 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:52:55.328 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:52:55.329 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:52:55.329 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:52:55.329 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:52:55.330 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:52:55.330 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:52:55.330 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:52:55.330 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:52:55.330 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:52:55.330 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:52:55.330 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:52:55.330 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:52:55.330 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:52:55.331 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:52:55.331 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T05:52:55.410 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:52:55.410 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:52:55.410 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:52:55.411 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:52:55.411 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:52:55.411 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:52:55.411 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:52:55.411 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:52:55.411 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:52:55.411 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:52:55.411 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:52:55.411 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:52:55.411 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:52:55.411 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T05:52:55.489 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:278: test_kill_daemon: run_mgr td/ceph-helpers x 2024-09-16T05:52:55.489 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T05:52:55.489 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T05:52:55.489 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T05:52:55.489 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T05:52:55.489 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T05:52:55.489 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T05:52:56.332 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T05:52:56.332 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:52:56.332 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:52:56.332 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:52:56.332 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:52:56.332 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:52:56.333 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:52:56.333 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:52:56.333 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:52:56.364 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:279: test_kill_daemon: run_osd td/ceph-helpers 0 2024-09-16T05:52:56.364 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T05:52:56.364 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:52:56.364 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T05:52:56.364 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:52:56.364 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T05:52:56.364 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T05:52:56.364 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:52:56.364 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:52:56.364 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:52:56.365 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T05:52:56.365 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T05:52:56.365 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:52:56.365 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:52:56.365 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T05:52:56.365 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:52:56.365 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:52:56.365 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:52:56.365 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:52:56.365 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:52:56.365 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:52:56.365 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:52:56.365 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:52:56.366 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:52:56.366 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:52:56.366 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:52:56.366 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T05:52:56.366 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T05:52:56.366 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:52:56.366 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:52:56.366 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:52:56.366 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:52:56.367 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:52:56.367 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:52:56.368 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T05:52:56.368 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:52:56.368 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=61efadc9-30be-4a3f-96cc-d5d23646c6c9 2024-09-16T05:52:56.369 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 61efadc9-30be-4a3f-96cc-d5d23646c6c9' 2024-09-16T05:52:56.369 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 61efadc9-30be-4a3f-96cc-d5d23646c6c9 2024-09-16T05:52:56.369 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:52:56.384 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC4x+dmOmzUFhAAZYoTPlRWHl0PXdDV9E9HBA== 2024-09-16T05:52:56.384 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC4x+dmOmzUFhAAZYoTPlRWHl0PXdDV9E9HBA=="}' 2024-09-16T05:52:56.384 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 61efadc9-30be-4a3f-96cc-d5d23646c6c9 -i td/ceph-helpers/0/new.json 2024-09-16T05:52:56.652 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:52:56.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T05:52:56.663 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC4x+dmOmzUFhAAZYoTPlRWHl0PXdDV9E9HBA== --osd-uuid 61efadc9-30be-4a3f-96cc-d5d23646c6c9 2024-09-16T05:52:56.693 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:52:56.692+0000 7fe81a218640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:52:56.721 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:52:56.720+0000 7fe81a218640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:52:56.723 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:52:56.722+0000 7fe81a218640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:52:56.723 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:52:56.722+0000 7fe81a218640 -1 bdev(0x5618d844bc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-16T05:52:56.723 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:52:56.722+0000 7fe81a218640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T05:53:02.467 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T05:53:02.468 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:53:02.468 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T05:53:02.468 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T05:53:02.469 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:53:02.980 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:53:02.980 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T05:53:02.981 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:53:02.981 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:53:02.981 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:53:02.981 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:53:03.007 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:53:03.006+0000 7f4559e0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:53:03.026 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:53:03.025+0000 7f4559e0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:53:03.027 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:53:03.026+0000 7f4559e0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:53:03.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T05:53:03.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:53:03.396 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:53:03.396 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:53:03.396 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:53:03.396 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:53:03.396 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:53:03.397 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:53:03.397 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:53:03.397 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:53:03.767 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:53:04.118 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:53:04.117+0000 7f4559e0f640 -1 Falling back to public interface 2024-09-16T05:53:04.768 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:53:04.768 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:53:04.768 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:53:04.769 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:53:04.769 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:53:04.769 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:53:05.147 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:53:06.148 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:53:06.148 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:53:06.149 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:53:06.149 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:53:06.149 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:53:06.149 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:53:06.528 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:53:07.320 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:53:07.320+0000 7f4559e0f640 -1 osd.0 0 log_to_monitors true 2024-09-16T05:53:07.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:53:07.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:53:07.530 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:53:07.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:53:07.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:53:07.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:53:08.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:53:09.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:53:09.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:53:09.041 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:53:09.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:53:09.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:53:09.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:53:09.454 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:53:10.455 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:53:10.455 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:53:10.456 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:53:10.456 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:53:10.456 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:53:10.456 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:53:10.864 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:53:11.865 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:53:11.866 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:53:11.866 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:53:11.866 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:53:11.866 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:53:11.867 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:53:12.399 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:53:13.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:53:13.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:53:13.401 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:53:13.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:53:13.402 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:53:13.402 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:53:13.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:53:14.821 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:53:14.821 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:53:14.821 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:53:14.822 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:53:14.822 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:53:14.822 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:53:15.257 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:53:16.258 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:53:16.258 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:53:16.259 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:53:16.259 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:53:16.259 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:53:16.259 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:53:16.654 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:53:17.655 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:53:17.656 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:53:17.656 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:53:17.656 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:53:17.657 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:53:17.657 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:53:18.185 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:53:19.186 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:53:19.186 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:53:19.187 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:53:19.187 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:53:19.187 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:53:19.187 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:53:19.756 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:53:20.757 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:53:20.757 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:53:20.758 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:53:20.758 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:53:20.758 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:53:20.758 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:53:21.336 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:53:22.337 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:53:22.337 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:53:22.338 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:53:22.338 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:53:22.338 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:53:22.338 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:53:23.259 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:53:24.261 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:53:24.261 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:53:24.261 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:53:24.261 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:53:24.262 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:53:24.262 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:53:24.712 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:53:25.712 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:53:25.712 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:53:25.712 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:53:25.713 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:53:25.713 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:53:25.713 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:53:26.427 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:53:27.428 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:53:27.428 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:53:27.429 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:53:27.429 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:53:27.429 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:53:27.429 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:53:28.156 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:53:29.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:53:29.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:53:29.158 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:53:29.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:53:29.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:53:29.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:53:30.008 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:53:31.010 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:53:31.010 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:53:31.010 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T05:53:31.011 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T05:53:31.011 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:53:31.011 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:53:31.388 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2509001146,v1:127.0.0.1:6803/2509001146] [v2:127.0.0.1:6804/2509001146,v1:127.0.0.1:6805/2509001146] exists,up 61efadc9-30be-4a3f-96cc-d5d23646c6c9 2024-09-16T05:53:31.389 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:53:31.390 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:53:31.390 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:53:31.390 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:281: test_kill_daemon: name_prefix=osd 2024-09-16T05:53:31.390 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:282: test_kill_daemon: find td/ceph-helpers 2024-09-16T05:53:31.390 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:282: test_kill_daemon: grep 'osd[^/]*\.pid' 2024-09-16T05:53:31.390 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:282: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2024-09-16T05:53:31.390 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:289: test_kill_daemon: kill_daemon td/ceph-helpers/osd.0.pid 0 1 2024-09-16T05:53:31.391 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/osd.0.pid 2024-09-16T05:53:31.392 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=124086 2024-09-16T05:53:31.392 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=0 2024-09-16T05:53:31.392 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local delays=1 2024-09-16T05:53:31.392 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2024-09-16T05:53:31.392 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-16T05:53:31.392 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 124086 2024-09-16T05:53:31.393 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-09-16T05:53:31.393 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-09-16T05:53:31.393 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 1 2024-09-16T05:53:32.393 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-16T05:53:32.393 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 124086 2024-09-16T05:53:32.393 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-09-16T05:53:32.393 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-09-16T05:53:32.393 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0 2024-09-16T05:53:32.394 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 1 2024-09-16T05:53:32.394 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:293: test_kill_daemon: kill_daemon td/ceph-helpers/osd.0.pid TERM 2024-09-16T05:53:32.394 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/osd.0.pid 2024-09-16T05:53:32.396 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=124086 2024-09-16T05:53:32.396 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2024-09-16T05:53:32.396 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2024-09-16T05:53:32.396 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2024-09-16T05:53:32.396 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-16T05:53:32.396 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 124086 2024-09-16T05:53:32.396 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-09-16T05:53:32.396 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-09-16T05:53:32.396 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2024-09-16T05:53:32.497 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-16T05:53:32.497 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 124086 2024-09-16T05:53:32.497 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-09-16T05:53:32.497 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-09-16T05:53:32.497 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.2 2024-09-16T05:53:32.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-16T05:53:32.699 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 124086 2024-09-16T05:53:32.699 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2024-09-16T05:53:32.699 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2024-09-16T05:53:32.699 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2024-09-16T05:53:32.699 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:296: test_kill_daemon: name_prefix=mgr 2024-09-16T05:53:32.699 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:297: test_kill_daemon: find td/ceph-helpers 2024-09-16T05:53:32.699 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:297: test_kill_daemon: grep 'mgr[^/]*\.pid' 2024-09-16T05:53:32.700 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:297: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2024-09-16T05:53:32.700 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:301: test_kill_daemon: kill_daemon td/ceph-helpers/mgr.x.pid TERM 2024-09-16T05:53:32.701 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/mgr.x.pid 2024-09-16T05:53:32.702 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=123136 2024-09-16T05:53:32.702 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2024-09-16T05:53:32.702 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2024-09-16T05:53:32.702 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2024-09-16T05:53:32.702 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-16T05:53:32.702 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 123136 2024-09-16T05:53:32.703 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-09-16T05:53:32.703 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-09-16T05:53:32.703 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2024-09-16T05:53:32.804 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-16T05:53:32.804 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 123136 2024-09-16T05:53:32.804 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2024-09-16T05:53:32.804 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2024-09-16T05:53:32.804 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2024-09-16T05:53:32.804 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:304: test_kill_daemon: name_prefix=mon 2024-09-16T05:53:32.804 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:305: test_kill_daemon: find td/ceph-helpers 2024-09-16T05:53:32.805 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:305: test_kill_daemon: grep 'mon[^/]*\.pid' 2024-09-16T05:53:32.806 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:305: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2024-09-16T05:53:32.806 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:309: test_kill_daemon: kill_daemon td/ceph-helpers/mon.a.pid TERM 2024-09-16T05:53:32.806 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/mon.a.pid 2024-09-16T05:53:32.807 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=123051 2024-09-16T05:53:32.807 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2024-09-16T05:53:32.808 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2024-09-16T05:53:32.808 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2024-09-16T05:53:32.808 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-16T05:53:32.808 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 123051 2024-09-16T05:53:32.808 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-09-16T05:53:32.808 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-09-16T05:53:32.808 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2024-09-16T05:53:32.909 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-16T05:53:32.909 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 123051 2024-09-16T05:53:32.909 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2024-09-16T05:53:32.909 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2024-09-16T05:53:32.909 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2024-09-16T05:53:32.909 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:310: test_kill_daemon: timeout 5 ceph status 2024-09-16T05:53:37.913 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:313: test_kill_daemon: teardown td/ceph-helpers 2024-09-16T05:53:37.913 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:53:37.913 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:53:37.913 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:53:37.913 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:53:37.914 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:53:37.914 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:53:37.914 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:53:37.914 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:53:37.918 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:53:37.918 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:53:37.919 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:53:37.919 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:53:37.921 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:53:37.921 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:53:37.921 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:53:37.922 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:53:37.922 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:53:37.922 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:53:37.922 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:53:37.923 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:53:37.924 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:53:37.925 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:53:37.931 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:53:37.931 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:53:37.931 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:53:37.931 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:53:37.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:53:37.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:53:37.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T05:53:37.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_kill_daemons td/ceph-helpers 2024-09-16T05:53:37.934 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:366: test_kill_daemons: local dir=td/ceph-helpers 2024-09-16T05:53:37.934 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:367: test_kill_daemons: setup td/ceph-helpers 2024-09-16T05:53:37.934 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T05:53:37.934 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T05:53:37.934 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:53:37.934 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:53:37.934 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:53:37.934 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:53:37.934 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:53:37.934 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:53:37.934 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:53:37.934 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:53:37.935 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:53:37.935 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:53:37.936 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:53:37.936 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:53:37.937 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:53:37.937 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:53:37.938 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:53:37.939 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:53:37.939 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:53:37.939 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:53:37.939 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:53:37.940 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:53:37.941 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:53:37.941 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:53:37.943 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:53:37.943 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:53:37.943 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:53:37.943 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:53:37.943 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:53:37.944 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:53:37.944 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T05:53:37.945 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:53:37.945 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:53:37.945 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:53:37.945 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T05:53:37.947 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:53:37.947 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:53:37.947 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:53:37.947 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T05:53:37.948 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:368: test_kill_daemons: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:53:37.948 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T05:53:37.948 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:53:37.948 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:53:37.948 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:53:37.948 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T05:53:37.948 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:53:38.305 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:53:38.306 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:53:38.306 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:53:38.306 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:53:38.306 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:53:38.306 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:53:38.306 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:53:38.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:53:38.506 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:53:38.509 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:53:38.509 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:53:38.509 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:53:38.509 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:53:38.510 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:53:38.510 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:53:38.510 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:53:38.510 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:53:38.510 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:53:38.510 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:53:38.511 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:53:38.511 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:53:38.511 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:53:38.511 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T05:53:38.599 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:53:38.599 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:53:38.599 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:53:38.599 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:53:38.600 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:53:38.600 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:53:38.600 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:53:38.600 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:53:38.600 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:53:38.600 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:53:38.601 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:53:38.601 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:53:38.601 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:53:38.601 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T05:53:38.678 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:369: test_kill_daemons: run_mgr td/ceph-helpers x 2024-09-16T05:53:38.678 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T05:53:38.678 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T05:53:38.679 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T05:53:38.679 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T05:53:38.679 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T05:53:38.679 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T05:53:39.519 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T05:53:39.519 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:53:39.519 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:53:39.519 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:53:39.519 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:53:39.519 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:53:39.519 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:53:39.520 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:53:39.520 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:53:39.552 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:370: test_kill_daemons: run_osd td/ceph-helpers 0 2024-09-16T05:53:39.552 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T05:53:39.552 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:53:39.552 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T05:53:39.552 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:53:39.552 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T05:53:39.552 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T05:53:39.552 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:53:39.552 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:53:39.552 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:53:39.552 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T05:53:39.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T05:53:39.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:53:39.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:53:39.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T05:53:39.553 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:53:39.553 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:53:39.553 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:53:39.553 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:53:39.553 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:53:39.553 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:53:39.553 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:53:39.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:53:39.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:53:39.554 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:53:39.554 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:53:39.554 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T05:53:39.554 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T05:53:39.554 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:53:39.554 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:53:39.554 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:53:39.554 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:53:39.554 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:53:39.554 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:53:39.554 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T05:53:39.554 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:53:39.554 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2f9c50b7-9d83-4732-a6f8-df41c46a48f5 2024-09-16T05:53:39.555 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2f9c50b7-9d83-4732-a6f8-df41c46a48f5' 2024-09-16T05:53:39.555 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 2f9c50b7-9d83-4732-a6f8-df41c46a48f5 2024-09-16T05:53:39.555 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:53:39.571 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDjx+dmLRb/IRAA1UudM7RbG1vJGYJWwUyEtg== 2024-09-16T05:53:39.571 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDjx+dmLRb/IRAA1UudM7RbG1vJGYJWwUyEtg=="}' 2024-09-16T05:53:39.571 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2f9c50b7-9d83-4732-a6f8-df41c46a48f5 -i td/ceph-helpers/0/new.json 2024-09-16T05:53:39.838 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:53:39.848 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T05:53:39.849 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDjx+dmLRb/IRAA1UudM7RbG1vJGYJWwUyEtg== --osd-uuid 2f9c50b7-9d83-4732-a6f8-df41c46a48f5 2024-09-16T05:53:39.878 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:53:39.878+0000 7ff3f180c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:53:39.907 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:53:39.906+0000 7ff3f180c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:53:39.908 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:53:39.907+0000 7ff3f180c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:53:39.909 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:53:39.908+0000 7ff3f180c640 -1 bdev(0x557ac71edc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-16T05:53:39.909 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:53:39.908+0000 7ff3f180c640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T05:53:45.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T05:53:45.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:53:45.482 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T05:53:45.483 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T05:53:45.483 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:53:46.024 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:53:46.025 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T05:53:46.025 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:53:46.025 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:53:46.025 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:53:46.025 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:53:46.051 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:53:46.051+0000 7f3aa50f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:53:46.068 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:53:46.067+0000 7f3aa50f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:53:46.069 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:53:46.068+0000 7f3aa50f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:53:46.464 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T05:53:46.464 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:53:46.464 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:53:46.464 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:53:46.465 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:53:46.465 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:53:46.465 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:53:46.465 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:53:46.466 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:53:46.466 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:53:46.833 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:53:47.418 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:53:47.418+0000 7f3aa50f1640 -1 Falling back to public interface 2024-09-16T05:53:47.835 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:53:47.835 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:53:47.835 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:53:47.836 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:53:47.836 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:53:47.836 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:53:48.221 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:53:49.222 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:53:49.223 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:53:49.223 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:53:49.223 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:53:49.223 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:53:49.223 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:53:49.610 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:53:50.197 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:53:50.195+0000 7f3aa50f1640 -1 osd.0 0 log_to_monitors true 2024-09-16T05:53:50.612 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:53:50.612 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:53:50.612 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:53:50.613 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:53:50.613 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:53:50.613 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:53:51.099 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:53:52.100 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:53:52.101 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:53:52.101 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:53:52.101 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:53:52.101 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:53:52.102 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:53:52.493 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:53:53.494 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:53:53.495 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:53:53.495 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:53:53.495 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:53:53.495 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:53:53.496 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:53:53.925 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:53:54.926 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:53:54.926 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:53:54.927 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:53:54.927 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:53:54.927 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:53:54.927 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:53:55.329 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:53:56.330 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:53:56.330 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:53:56.330 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:53:56.331 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:53:56.331 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:53:56.331 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:53:56.709 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:53:57.711 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:53:57.711 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:53:57.711 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:53:57.712 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:53:57.712 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:53:57.712 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:53:58.131 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:53:59.132 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:53:59.132 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:53:59.133 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:53:59.133 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:53:59.133 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:53:59.133 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:53:59.514 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:54:00.515 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:54:00.515 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:54:00.515 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:54:00.516 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:54:00.516 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:54:00.516 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:54:00.942 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:54:01.943 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:54:01.943 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:54:01.943 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:54:01.944 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:54:01.944 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:54:01.944 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:54:02.647 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:54:03.648 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:54:03.649 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:54:03.649 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:54:03.649 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:54:03.649 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:54:03.650 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:54:04.463 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:54:05.465 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:54:05.465 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:54:05.465 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:54:05.466 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:54:05.466 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:54:05.466 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:54:05.969 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:54:06.970 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:54:06.971 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:54:06.971 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:54:06.972 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:54:06.972 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:54:06.972 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:54:07.688 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:54:08.689 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:54:08.689 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:54:08.690 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:54:08.690 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:54:08.690 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:54:08.690 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:54:09.356 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:54:10.357 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:54:10.357 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:54:10.358 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:54:10.358 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:54:10.358 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:54:10.358 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:54:11.033 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:54:12.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:54:12.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:54:12.034 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:54:12.035 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:54:12.035 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:54:12.035 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:54:12.692 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:54:13.694 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:54:13.694 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:54:13.694 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T05:54:13.694 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T05:54:13.695 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:54:13.695 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:54:14.115 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/890903818,v1:127.0.0.1:6803/890903818] [v2:127.0.0.1:6804/890903818,v1:127.0.0.1:6805/890903818] exists,up 2f9c50b7-9d83-4732-a6f8-df41c46a48f5 2024-09-16T05:54:14.116 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:54:14.116 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:54:14.116 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:54:14.116 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:377: test_kill_daemons: kill_daemons td/ceph-helpers 0 osd 1 2024-09-16T05:54:14.116 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:54:14.116 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:54:14.116 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:54:14.116 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:54:14.116 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:54:15.122 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 1 2024-09-16T05:54:15.122 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:381: test_kill_daemons: kill_daemons td/ceph-helpers TERM osd 2024-09-16T05:54:15.122 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:54:15.122 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:54:15.123 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:54:15.123 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:54:15.123 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:54:15.429 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:54:15.429 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:385: test_kill_daemons: kill_daemons td/ceph-helpers TERM mgr 2024-09-16T05:54:15.430 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:54:15.430 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:54:15.430 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:54:15.430 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:54:15.430 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:54:15.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:54:15.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:389: test_kill_daemons: kill_daemons td/ceph-helpers TERM 2024-09-16T05:54:15.537 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:54:15.537 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:54:15.537 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:54:15.537 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:54:15.537 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:54:15.844 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:54:15.844 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:390: test_kill_daemons: timeout 5 ceph status 2024-09-16T05:54:20.848 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:391: test_kill_daemons: teardown td/ceph-helpers 2024-09-16T05:54:20.849 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:54:20.849 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:54:20.849 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:54:20.849 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:54:20.849 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:54:20.849 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:54:20.849 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:54:20.849 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:54:20.854 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:54:20.854 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:54:20.855 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:54:20.856 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:54:20.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:54:20.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:54:20.857 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:54:20.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:54:20.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:54:20.859 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:54:20.859 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:54:20.859 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:54:20.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:54:20.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:54:20.867 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:54:20.867 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:54:20.867 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:54:20.867 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:54:20.869 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:54:20.869 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:54:20.870 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T05:54:20.870 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_objectstore_tool td/ceph-helpers 2024-09-16T05:54:20.870 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: test_objectstore_tool: local dir=td/ceph-helpers 2024-09-16T05:54:20.870 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: test_objectstore_tool: setup td/ceph-helpers 2024-09-16T05:54:20.870 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T05:54:20.870 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T05:54:20.870 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:54:20.870 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:54:20.870 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:54:20.870 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:54:20.870 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:54:20.870 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:54:20.870 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:54:20.870 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:54:20.871 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:54:20.871 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:54:20.872 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:54:20.872 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:54:20.873 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:54:20.873 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:54:20.874 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:54:20.875 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:54:20.875 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:54:20.875 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:54:20.875 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:54:20.876 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:54:20.877 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:54:20.877 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:54:20.878 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:54:20.879 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:54:20.879 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:54:20.879 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:54:20.880 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:54:20.880 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:54:20.880 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T05:54:20.881 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:54:20.881 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:54:20.881 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:54:20.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T05:54:20.883 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:54:20.883 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:54:20.883 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:54:20.883 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T05:54:20.884 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: test_objectstore_tool: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:54:20.884 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T05:54:20.884 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:54:20.884 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:54:20.884 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:54:20.884 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T05:54:20.884 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:54:21.261 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:54:21.262 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:54:21.262 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:54:21.262 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:54:21.262 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:54:21.262 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:54:21.263 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:54:21.263 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:54:21.460 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:54:21.462 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:54:21.462 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:54:21.462 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:54:21.462 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:54:21.462 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:54:21.462 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:54:21.462 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:54:21.462 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:54:21.463 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:54:21.463 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:54:21.463 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:54:21.463 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:54:21.463 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:54:21.463 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T05:54:21.544 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:54:21.545 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:54:21.545 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:54:21.545 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:54:21.545 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:54:21.545 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:54:21.545 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:54:21.545 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:54:21.545 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:54:21.545 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:54:21.545 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:54:21.545 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:54:21.545 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:54:21.545 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T05:54:21.623 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1311: test_objectstore_tool: local osd=0 2024-09-16T05:54:21.623 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: test_objectstore_tool: run_mgr td/ceph-helpers x 2024-09-16T05:54:21.623 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T05:54:21.623 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T05:54:21.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T05:54:21.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T05:54:21.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T05:54:21.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T05:54:22.464 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T05:54:22.464 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:54:22.464 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:54:22.464 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:54:22.464 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:54:22.464 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:54:22.464 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:54:22.465 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:54:22.465 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:54:22.495 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: test_objectstore_tool: run_osd td/ceph-helpers 0 2024-09-16T05:54:22.496 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T05:54:22.496 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:54:22.496 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T05:54:22.496 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:54:22.496 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T05:54:22.496 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T05:54:22.496 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:54:22.496 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:54:22.496 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:54:22.496 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T05:54:22.496 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T05:54:22.497 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:54:22.497 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:54:22.497 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T05:54:22.497 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:54:22.497 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:54:22.497 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:54:22.497 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:54:22.497 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:54:22.497 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:54:22.497 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:54:22.497 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:54:22.497 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:54:22.497 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:54:22.497 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:54:22.498 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T05:54:22.498 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T05:54:22.498 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:54:22.498 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:54:22.498 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:54:22.498 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:54:22.498 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:54:22.498 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:54:22.498 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T05:54:22.498 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:54:22.499 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b72c1ffe-1d48-40db-93b8-8f9f4aac392a 2024-09-16T05:54:22.499 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b72c1ffe-1d48-40db-93b8-8f9f4aac392a' 2024-09-16T05:54:22.499 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 b72c1ffe-1d48-40db-93b8-8f9f4aac392a 2024-09-16T05:54:22.499 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:54:22.515 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAOyOdmH8izHhAADdSrBPflgQukA6HHnJnKrA== 2024-09-16T05:54:22.516 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAOyOdmH8izHhAADdSrBPflgQukA6HHnJnKrA=="}' 2024-09-16T05:54:22.516 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b72c1ffe-1d48-40db-93b8-8f9f4aac392a -i td/ceph-helpers/0/new.json 2024-09-16T05:54:22.793 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:54:22.803 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T05:54:22.804 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAOyOdmH8izHhAADdSrBPflgQukA6HHnJnKrA== --osd-uuid b72c1ffe-1d48-40db-93b8-8f9f4aac392a 2024-09-16T05:54:22.833 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:54:22.832+0000 7fcddff56640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:54:22.862 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:54:22.862+0000 7fcddff56640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:54:22.863 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:54:22.863+0000 7fcddff56640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:54:22.864 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:54:22.863+0000 7fcddff56640 -1 bdev(0x560d24bebc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-16T05:54:22.864 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:54:22.864+0000 7fcddff56640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T05:54:28.550 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T05:54:28.550 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:54:28.551 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T05:54:28.552 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T05:54:28.552 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:54:29.069 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:54:29.070 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T05:54:29.070 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:54:29.070 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:54:29.070 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:54:29.070 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:54:29.097 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:54:29.096+0000 7fae3f5cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:54:29.122 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:54:29.121+0000 7fae3f5cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:54:29.123 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:54:29.122+0000 7fae3f5cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:54:29.444 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T05:54:29.444 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:54:29.445 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:54:29.445 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:54:29.445 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:54:29.445 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:54:29.445 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:54:29.446 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:54:29.446 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:54:29.446 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:54:29.817 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:54:29.993 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:54:29.993+0000 7fae3f5cc640 -1 Falling back to public interface 2024-09-16T05:54:30.818 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:54:30.818 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:54:30.818 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:54:30.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:54:30.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:54:30.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:54:31.236 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:54:32.238 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:54:32.238 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:54:32.238 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:54:32.239 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:54:32.239 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:54:32.239 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:54:32.623 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:54:33.066 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:54:33.065+0000 7fae3f5cc640 -1 osd.0 0 log_to_monitors true 2024-09-16T05:54:33.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:54:33.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:54:33.624 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:54:33.625 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:54:33.625 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:54:33.625 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:54:34.047 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:54:35.048 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:54:35.048 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:54:35.049 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:54:35.049 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:54:35.049 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:54:35.049 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:54:35.456 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:54:36.457 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:54:36.458 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:54:36.458 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:54:36.458 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:54:36.459 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:54:36.459 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:54:36.945 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:54:37.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:54:37.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:54:37.946 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:54:37.947 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:54:37.947 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:54:37.947 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:54:38.369 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:54:39.370 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:54:39.370 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:54:39.371 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:54:39.371 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:54:39.371 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:54:39.371 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:54:39.841 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:54:40.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:54:40.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:54:40.842 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:54:40.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:54:40.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:54:40.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:54:41.228 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:54:42.229 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:54:42.229 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:54:42.229 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:54:42.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:54:42.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:54:42.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:54:42.611 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:54:43.613 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:54:43.613 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:54:43.613 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:54:43.614 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:54:43.614 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:54:43.614 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:54:44.162 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:54:45.164 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:54:45.164 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:54:45.164 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:54:45.165 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:54:45.165 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:54:45.165 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:54:45.583 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:54:46.584 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:54:46.584 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:54:46.585 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:54:46.585 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:54:46.585 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:54:46.585 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:54:47.405 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:54:48.412 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:54:48.412 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:54:48.413 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:54:48.413 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:54:48.413 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:54:48.413 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:54:48.863 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:54:49.864 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:54:49.865 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:54:49.865 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:54:49.865 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:54:49.865 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:54:49.866 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:54:50.602 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:54:51.603 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:54:51.604 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:54:51.604 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:54:51.605 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:54:51.605 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:54:51.605 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:54:52.413 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:54:53.414 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:54:53.414 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:54:53.415 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:54:53.415 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:54:53.415 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:54:53.416 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:54:54.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:54:55.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:54:55.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:54:55.159 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:54:55.160 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:54:55.160 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:54:55.160 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:54:55.909 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:54:56.910 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:54:56.910 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:54:56.911 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T05:54:56.911 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T05:54:56.911 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:54:56.911 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:54:57.293 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2220275601,v1:127.0.0.1:6803/2220275601] [v2:127.0.0.1:6804/2220275601,v1:127.0.0.1:6805/2220275601] exists,up b72c1ffe-1d48-40db-93b8-8f9f4aac392a 2024-09-16T05:54:57.294 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:54:57.294 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:54:57.294 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:54:57.295 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: test_objectstore_tool: create_rbd_pool 2024-09-16T05:54:57.295 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T05:54:57.659 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' does not exist 2024-09-16T05:54:57.670 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T05:54:57.670 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T05:54:58.188 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' already exists 2024-09-16T05:54:58.199 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T05:54:59.201 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T05:55:00.257 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: test_objectstore_tool: wait_for_clean 2024-09-16T05:55:00.257 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T05:55:00.257 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T05:55:00.257 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T05:55:00.257 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T05:55:00.258 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T05:55:00.258 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T05:55:00.258 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T05:55:00.258 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T05:55:00.258 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T05:55:00.354 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T05:55:00.354 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T05:55:00.355 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T05:55:00.355 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T05:55:00.355 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-16T05:55:00.355 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-16T05:55:00.731 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-09-16T05:55:00.731 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-16T05:55:00.731 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:55:00.731 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T05:55:00.844 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836488 2024-09-16T05:55:00.844 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836488 2024-09-16T05:55:00.844 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836488' 2024-09-16T05:55:00.844 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:55:00.845 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836488 2024-09-16T05:55:00.845 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:55:00.845 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-16T05:55:00.846 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836488 2024-09-16T05:55:00.846 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:55:00.847 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836488 2024-09-16T05:55:00.847 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2024-09-16T05:55:00.847 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.0 seq 21474836488 2024-09-16T05:55:00.847 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:55:01.261 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836486 -lt 21474836488 2024-09-16T05:55:01.261 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-16T05:55:02.262 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T05:55:02.263 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:55:02.653 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836489 -lt 21474836488 2024-09-16T05:55:02.654 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T05:55:02.654 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:55:02.654 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:55:03.161 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T05:55:03.161 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T05:55:03.162 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T05:55:03.162 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T05:55:03.162 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T05:55:03.162 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T05:55:03.162 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T05:55:03.162 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T05:55:03.526 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T05:55:03.526 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T05:55:03.526 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:55:03.526 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:55:04.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T05:55:04.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T05:55:04.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T05:55:04.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1316: test_objectstore_tool: rados --pool rbd put GROUP /etc/group 2024-09-16T05:55:04.205 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: test_objectstore_tool: objectstore_tool td/ceph-helpers 0 GROUP get-bytes 2024-09-16T05:55:04.206 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/ceph-helpers 2024-09-16T05:55:04.206 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: test_objectstore_tool: diff - /etc/group 2024-09-16T05:55:04.206 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-16T05:55:04.206 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-16T05:55:04.206 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-16T05:55:04.206 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/ceph-helpers 0 GROUP get-bytes 2024-09-16T05:55:04.206 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/ceph-helpers 2024-09-16T05:55:04.206 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-16T05:55:04.206 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-16T05:55:04.206 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-16T05:55:04.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/ceph-helpers TERM osd.0 2024-09-16T05:55:04.207 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:55:04.207 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:55:04.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:55:04.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:55:04.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:55:04.511 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:55:04.512 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/ceph-helpers 0 GROUP get-bytes 2024-09-16T05:55:04.512 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/ceph-helpers 2024-09-16T05:55:04.512 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-16T05:55:04.512 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-16T05:55:04.512 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-16T05:55:04.512 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/ceph-helpers/0 2024-09-16T05:55:04.512 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/ceph-helpers/0 GROUP get-bytes 2024-09-16T05:55:06.133 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/ceph-helpers 0 2024-09-16T05:55:06.133 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-09-16T05:55:06.133 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T05:55:06.133 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T05:55:06.133 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T05:55:06.133 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-09-16T05:55:06.134 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T05:55:06.134 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:55:06.134 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:55:06.134 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:55:06.134 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T05:55:06.134 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T05:55:06.134 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T05:55:06.134 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T05:55:06.134 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T05:55:06.134 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T05:55:06.135 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:55:06.135 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:55:06.135 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:55:06.135 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:55:06.135 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:55:06.135 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:55:06.135 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:55:06.135 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:55:06.135 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T05:55:06.135 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T05:55:06.135 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:55:06.135 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:55:06.135 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:55:06.135 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:55:06.136 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T05:55:06.136 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T05:55:06.136 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-09-16T05:55:06.137 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T05:55:06.137 INFO:tasks.workunit.client.0.smithi178.stderr:start osd.0 2024-09-16T05:55:06.137 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:55:06.137 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-09-16T05:55:06.137 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T05:55:06.137 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T05:55:06.137 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T05:55:06.137 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T05:55:06.164 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:55:06.164+0000 7fcfbc815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:55:06.166 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:55:06.166+0000 7fcfbc815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:55:06.168 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:55:06.167+0000 7fcfbc815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:55:06.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T05:55:06.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:55:06.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:55:06.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:55:06.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:55:06.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:55:06.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:55:06.536 INFO:tasks.workunit.client.0.smithi178.stderr:0 2024-09-16T05:55:06.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:55:06.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:55:06.931 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:55:07.018 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:55:07.018+0000 7fcfbc815640 -1 Falling back to public interface 2024-09-16T05:55:07.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:55:07.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:55:07.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:55:07.933 INFO:tasks.workunit.client.0.smithi178.stderr:1 2024-09-16T05:55:07.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:55:07.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:55:08.310 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:55:09.312 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:55:09.312 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:55:09.312 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:55:09.312 INFO:tasks.workunit.client.0.smithi178.stderr:2 2024-09-16T05:55:09.312 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:55:09.312 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:55:09.595 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:55:09.595+0000 7fcfbc815640 -1 osd.0 12 log_to_monitors true 2024-09-16T05:55:09.694 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:55:10.695 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:55:10.695 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:55:10.695 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:55:10.695 INFO:tasks.workunit.client.0.smithi178.stderr:3 2024-09-16T05:55:10.695 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:55:10.695 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:55:11.110 INFO:tasks.workunit.client.0.smithi178.stderr:osd.0 up in weight 1 up_from 16 up_thru 16 down_at 13 last_clean_interval [5,12) [v2:127.0.0.1:6802/1542951472,v1:127.0.0.1:6803/1542951472] [v2:127.0.0.1:6804/1542951472,v1:127.0.0.1:6805/1542951472] exists,up b72c1ffe-1d48-40db-93b8-8f9f4aac392a 2024-09-16T05:55:11.111 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:55:11.111 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:55:11.111 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:55:11.111 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-16T05:55:11.111 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T05:55:11.111 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T05:55:11.111 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T05:55:11.111 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T05:55:11.111 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T05:55:11.111 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T05:55:11.111 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T05:55:11.111 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T05:55:11.111 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T05:55:11.193 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T05:55:11.193 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T05:55:11.193 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T05:55:11.193 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T05:55:11.193 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-16T05:55:11.193 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-16T05:55:11.595 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-09-16T05:55:11.595 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-16T05:55:11.595 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:55:11.595 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T05:55:11.717 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=68719476739 2024-09-16T05:55:11.717 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 68719476739 2024-09-16T05:55:11.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-68719476739' 2024-09-16T05:55:11.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:55:11.718 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-68719476739 2024-09-16T05:55:11.718 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:55:11.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-16T05:55:11.719 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-68719476739 2024-09-16T05:55:11.719 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:55:11.720 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=68719476739 2024-09-16T05:55:11.720 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 68719476739' 2024-09-16T05:55:11.720 INFO:tasks.workunit.client.0.smithi178.stderr:waiting osd.0 seq 68719476739 2024-09-16T05:55:11.720 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:55:12.087 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 68719476739 2024-09-16T05:55:12.087 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-16T05:55:13.088 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T05:55:13.088 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:55:13.463 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 68719476739 -lt 68719476739 2024-09-16T05:55:13.464 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T05:55:13.464 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:55:13.464 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:55:13.967 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T05:55:13.967 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T05:55:13.967 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T05:55:13.967 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T05:55:13.968 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T05:55:13.968 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T05:55:13.968 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T05:55:13.968 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T05:55:14.376 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T05:55:14.376 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T05:55:14.376 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:55:14.376 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:55:14.883 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T05:55:14.883 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T05:55:14.883 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T05:55:14.884 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1319: test_objectstore_tool: objectstore_tool td/ceph-helpers 0 NOTEXISTS get-bytes 2024-09-16T05:55:14.884 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/ceph-helpers 2024-09-16T05:55:14.884 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-16T05:55:14.884 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-16T05:55:14.884 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-16T05:55:14.884 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/ceph-helpers 0 NOTEXISTS get-bytes 2024-09-16T05:55:14.885 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/ceph-helpers 2024-09-16T05:55:14.885 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-16T05:55:14.885 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-16T05:55:14.885 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-16T05:55:14.885 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/ceph-helpers TERM osd.0 2024-09-16T05:55:14.885 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:55:14.886 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:55:14.886 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:55:14.886 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:55:14.886 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:55:15.187 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:55:15.188 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/ceph-helpers 0 NOTEXISTS get-bytes 2024-09-16T05:55:15.188 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/ceph-helpers 2024-09-16T05:55:15.188 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-16T05:55:15.188 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-16T05:55:15.188 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-16T05:55:15.188 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/ceph-helpers/0 2024-09-16T05:55:15.188 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/ceph-helpers/0 NOTEXISTS get-bytes 2024-09-16T05:55:16.568 INFO:tasks.workunit.client.0.smithi178.stderr:No object id 'NOTEXISTS' found or invalid JSON specified 2024-09-16T05:55:16.835 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-09-16T05:55:16.836 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: return 1 2024-09-16T05:55:16.836 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: return 1 2024-09-16T05:55:16.836 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1320: test_objectstore_tool: teardown td/ceph-helpers 2024-09-16T05:55:16.836 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:55:16.836 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:55:16.837 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:55:16.837 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:55:16.837 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:55:16.837 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:55:16.837 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:55:16.837 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:55:16.942 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:55:16.943 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:55:16.943 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:55:16.944 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:55:16.945 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:55:16.945 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:55:16.945 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:55:16.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:55:16.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:55:16.947 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:55:16.947 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:55:16.947 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:55:16.949 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:55:16.949 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:55:16.956 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:55:16.956 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:55:16.956 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:55:16.956 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:55:16.958 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:55:16.958 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:55:16.958 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T05:55:16.958 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_pg_schedule_scrub td/ceph-helpers 2024-09-16T05:55:16.958 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1931: test_pg_schedule_scrub: local dir=td/ceph-helpers 2024-09-16T05:55:16.959 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1933: test_pg_schedule_scrub: setup td/ceph-helpers 2024-09-16T05:55:16.959 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T05:55:16.959 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T05:55:16.959 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:55:16.959 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:55:16.959 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:55:16.959 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:55:16.959 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:55:16.959 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:55:16.959 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:55:16.959 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:55:16.960 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:55:16.960 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:55:16.961 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:55:16.961 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:55:16.962 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:55:16.962 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:55:16.962 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:55:16.963 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:55:16.963 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:55:16.964 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:55:16.964 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:55:16.964 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:55:16.965 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:55:16.966 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:55:16.967 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:55:16.967 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:55:16.967 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:55:16.967 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:55:16.968 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:55:16.968 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:55:16.968 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T05:55:16.969 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:55:16.969 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:55:16.970 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:55:16.970 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T05:55:16.972 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:55:16.972 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:55:16.972 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:55:16.972 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T05:55:16.972 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: test_pg_schedule_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:55:16.972 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T05:55:16.972 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:55:16.972 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:55:16.972 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:55:16.972 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T05:55:16.972 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:55:17.351 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:55:17.351 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:55:17.351 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:55:17.351 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:55:17.352 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:55:17.352 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:55:17.352 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:55:17.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:55:17.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:55:17.544 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:55:17.544 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:55:17.544 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:55:17.544 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:55:17.544 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:55:17.545 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:55:17.545 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:55:17.546 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:55:17.546 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:55:17.546 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:55:17.546 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:55:17.546 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:55:17.546 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:55:17.546 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T05:55:17.628 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:55:17.628 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:55:17.628 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:55:17.629 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:55:17.629 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:55:17.629 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:55:17.629 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:55:17.629 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:55:17.629 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:55:17.629 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:55:17.629 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:55:17.629 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:55:17.629 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:55:17.629 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T05:55:17.707 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: test_pg_schedule_scrub: run_mgr td/ceph-helpers x 2024-09-16T05:55:17.707 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T05:55:17.707 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T05:55:17.707 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T05:55:17.707 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T05:55:17.707 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T05:55:17.707 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T05:55:18.555 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T05:55:18.556 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:55:18.556 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:55:18.556 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:55:18.556 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:55:18.556 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:55:18.556 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:55:18.557 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:55:18.557 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:55:18.587 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: test_pg_schedule_scrub: run_osd td/ceph-helpers 0 2024-09-16T05:55:18.587 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T05:55:18.587 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:55:18.587 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T05:55:18.587 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:55:18.587 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T05:55:18.587 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T05:55:18.588 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:55:18.588 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:55:18.588 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:55:18.588 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T05:55:18.588 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T05:55:18.588 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:55:18.588 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:55:18.588 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T05:55:18.588 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:55:18.588 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:55:18.588 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:55:18.588 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:55:18.589 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:55:18.589 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:55:18.589 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:55:18.589 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:55:18.589 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:55:18.589 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:55:18.589 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:55:18.589 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T05:55:18.589 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T05:55:18.589 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:55:18.589 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:55:18.589 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:55:18.589 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:55:18.589 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:55:18.590 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:55:18.590 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T05:55:18.590 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:55:18.590 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5fae2c81-ba64-4928-8a80-9ea953b87d7d 2024-09-16T05:55:18.590 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 5fae2c81-ba64-4928-8a80-9ea953b87d7d 2024-09-16T05:55:18.591 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 5fae2c81-ba64-4928-8a80-9ea953b87d7d' 2024-09-16T05:55:18.591 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:55:18.607 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBGyOdmhnQkJBAALoQLPlrYLAzjL8HuFl6Qdg== 2024-09-16T05:55:18.607 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBGyOdmhnQkJBAALoQLPlrYLAzjL8HuFl6Qdg=="}' 2024-09-16T05:55:18.607 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5fae2c81-ba64-4928-8a80-9ea953b87d7d -i td/ceph-helpers/0/new.json 2024-09-16T05:55:18.886 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:55:18.899 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T05:55:18.900 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBGyOdmhnQkJBAALoQLPlrYLAzjL8HuFl6Qdg== --osd-uuid 5fae2c81-ba64-4928-8a80-9ea953b87d7d 2024-09-16T05:55:18.924 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:55:18.923+0000 7f0a13c15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:55:18.953 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:55:18.953+0000 7f0a13c15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:55:18.954 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:55:18.954+0000 7f0a13c15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:55:18.954 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:55:18.954+0000 7f0a13c15640 -1 bdev(0x56464f965c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-16T05:55:18.955 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:55:18.954+0000 7f0a13c15640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T05:55:24.666 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T05:55:24.667 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:55:24.667 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T05:55:24.667 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T05:55:24.667 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:55:25.190 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:55:25.191 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T05:55:25.191 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:55:25.191 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:55:25.191 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:55:25.191 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:55:25.217 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:55:25.216+0000 7f18cd00b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:55:25.233 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:55:25.232+0000 7f18cd00b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:55:25.234 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:55:25.233+0000 7f18cd00b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:55:25.638 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T05:55:25.638 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:55:25.638 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:55:25.638 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:55:25.638 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:55:25.638 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:55:25.638 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:55:25.639 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:55:25.639 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:55:25.639 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:55:26.005 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:55:26.823 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:55:26.823+0000 7f18cd00b640 -1 Falling back to public interface 2024-09-16T05:55:27.007 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:55:27.007 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:55:27.007 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:55:27.008 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:55:27.008 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:55:27.008 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:55:27.387 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:55:28.389 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:55:28.389 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:55:28.389 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:55:28.390 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:55:28.390 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:55:28.390 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:55:28.808 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:55:29.519 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:55:29.519+0000 7f18cd00b640 -1 osd.0 0 log_to_monitors true 2024-09-16T05:55:29.809 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:55:29.809 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:55:29.809 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:55:29.810 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:55:29.810 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:55:29.810 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:55:30.295 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:55:31.296 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:55:31.297 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:55:31.297 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:55:31.297 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:55:31.298 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:55:31.298 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:55:31.691 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:55:32.693 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:55:32.693 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:55:32.693 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:55:32.694 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:55:32.694 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:55:32.694 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:55:33.078 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:55:34.079 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:55:34.080 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:55:34.080 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:55:34.080 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:55:34.081 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:55:34.081 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:55:34.634 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:55:35.635 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:55:35.635 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:55:35.635 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:55:35.636 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:55:35.636 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:55:35.636 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:55:36.046 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:55:37.047 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:55:37.048 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:55:37.048 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:55:37.048 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:55:37.049 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:55:37.049 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:55:37.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:55:38.442 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:55:38.442 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:55:38.442 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:55:38.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:55:38.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:55:38.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:55:38.823 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:55:39.825 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:55:39.825 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:55:39.825 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:55:39.826 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:55:39.826 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:55:39.826 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:55:40.330 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:55:41.332 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:55:41.332 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:55:41.332 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:55:41.333 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:55:41.333 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:55:41.333 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:55:41.782 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:55:42.783 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:55:42.783 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:55:42.783 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:55:42.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:55:42.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:55:42.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:55:43.603 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:55:44.604 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:55:44.604 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:55:44.605 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:55:44.605 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:55:44.605 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:55:44.605 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:55:45.449 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:55:46.450 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:55:46.450 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:55:46.451 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:55:46.451 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:55:46.451 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:55:46.451 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:55:46.863 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:55:47.864 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:55:47.865 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:55:47.865 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:55:47.865 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:55:47.865 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:55:47.865 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:55:48.439 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:55:49.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:55:49.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:55:49.441 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:55:49.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:55:49.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:55:49.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:55:50.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:55:51.056 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:55:51.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:55:51.057 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:55:51.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:55:51.058 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:55:51.058 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:55:52.083 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:55:53.085 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:55:53.085 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:55:53.085 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T05:55:53.086 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T05:55:53.086 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:55:53.086 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:55:53.527 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/976474896,v1:127.0.0.1:6803/976474896] [v2:127.0.0.1:6804/976474896,v1:127.0.0.1:6805/976474896] exists,up 5fae2c81-ba64-4928-8a80-9ea953b87d7d 2024-09-16T05:55:53.528 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:55:53.528 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:55:53.528 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:55:53.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: test_pg_schedule_scrub: create_rbd_pool 2024-09-16T05:55:53.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T05:55:53.907 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' does not exist 2024-09-16T05:55:53.919 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T05:55:53.919 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T05:55:54.389 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' already exists 2024-09-16T05:55:54.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T05:55:55.402 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T05:55:56.466 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1938: test_pg_schedule_scrub: wait_for_clean 2024-09-16T05:55:56.466 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T05:55:56.466 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T05:55:56.466 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T05:55:56.467 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T05:55:56.467 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T05:55:56.467 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T05:55:56.467 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T05:55:56.467 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T05:55:56.468 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T05:55:56.563 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T05:55:56.563 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T05:55:56.563 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T05:55:56.563 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T05:55:56.563 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-16T05:55:56.563 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-16T05:55:56.960 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-09-16T05:55:56.960 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-16T05:55:56.961 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:55:56.961 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T05:55:57.084 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836483 2024-09-16T05:55:57.084 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836483 2024-09-16T05:55:57.084 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836483' 2024-09-16T05:55:57.084 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:55:57.084 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836483 2024-09-16T05:55:57.084 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:55:57.085 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-16T05:55:57.085 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836483 2024-09-16T05:55:57.086 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:55:57.086 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836483 2024-09-16T05:55:57.087 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.0 seq 21474836483 2024-09-16T05:55:57.087 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2024-09-16T05:55:57.087 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:55:57.453 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836481 -lt 21474836483 2024-09-16T05:55:57.454 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-16T05:55:58.455 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T05:55:58.455 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:55:58.863 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836484 -lt 21474836483 2024-09-16T05:55:58.864 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T05:55:58.864 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:55:58.864 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:55:59.368 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T05:55:59.368 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T05:55:59.368 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T05:55:59.369 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T05:55:59.369 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T05:55:59.369 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T05:55:59.369 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T05:55:59.369 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T05:55:59.767 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T05:55:59.767 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T05:55:59.767 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:55:59.768 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:56:00.267 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T05:56:00.267 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T05:56:00.267 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T05:56:00.267 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1939: test_pg_schedule_scrub: pg_schedule_scrub 1.0 2024-09-16T05:56:00.267 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1917: pg_schedule_scrub: local pgid=1.0 2024-09-16T05:56:00.267 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: pg_schedule_scrub: get_last_scrub_stamp 1.0 2024-09-16T05:56:00.267 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T05:56:00.268 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T05:56:00.268 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T05:56:00.268 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T05:56:00.666 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: pg_schedule_scrub: local last_scrub=2024-09-16T05:55:54.233201+0000 2024-09-16T05:56:00.666 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1919: pg_schedule_scrub: ceph pg scrub 1.0 2024-09-16T05:56:01.018 INFO:tasks.workunit.client.0.smithi178.stderr:instructing pg 1.0 on osd.0 to scrub 2024-09-16T05:56:01.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1920: pg_schedule_scrub: wait_for_scrub 1.0 2024-09-16T05:55:54.233201+0000 2024-09-16T05:56:01.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-09-16T05:56:01.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-09-16T05:55:54.233201+0000 2024-09-16T05:56:01.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-09-16T05:56:01.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-09-16T05:56:01.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-09-16T05:56:01.031 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T05:56:01.031 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T05:56:01.031 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T05:56:01.031 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T05:56:01.031 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T05:56:01.389 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-09-16T05:55:54.233201+0000 '>' 2024-09-16T05:55:54.233201+0000 2024-09-16T05:56:01.390 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-09-16T05:56:02.391 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-09-16T05:56:02.391 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-09-16T05:56:02.392 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T05:56:02.392 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T05:56:02.392 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T05:56:02.392 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T05:56:02.392 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T05:56:02.769 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-09-16T05:55:54.233201+0000 '>' 2024-09-16T05:55:54.233201+0000 2024-09-16T05:56:02.769 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-09-16T05:56:03.770 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-09-16T05:56:03.770 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-09-16T05:56:03.770 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T05:56:03.771 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T05:56:03.771 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T05:56:03.771 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T05:56:03.771 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T05:56:04.144 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-09-16T05:55:54.233201+0000 '>' 2024-09-16T05:55:54.233201+0000 2024-09-16T05:56:04.145 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-09-16T05:56:05.146 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-09-16T05:56:05.147 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-09-16T05:56:05.147 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T05:56:05.147 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T05:56:05.147 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T05:56:05.148 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T05:56:05.148 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T05:56:05.517 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-09-16T05:56:01.381559+0000 '>' 2024-09-16T05:55:54.233201+0000 2024-09-16T05:56:05.518 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-09-16T05:56:05.518 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1940: test_pg_schedule_scrub: kill_daemons td/ceph-helpers KILL osd 2024-09-16T05:56:05.518 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:56:05.518 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:56:05.518 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:56:05.519 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:56:05.519 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:56:05.623 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:56:05.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1941: test_pg_schedule_scrub: TIMEOUT=1 2024-09-16T05:56:05.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1941: test_pg_schedule_scrub: pg_scrub 1.0 2024-09-16T05:56:05.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=1.0 2024-09-16T05:56:05.624 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 1.0 2024-09-16T05:56:05.624 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T05:56:05.625 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T05:56:05.625 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T05:56:05.625 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T05:56:06.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-09-16T05:56:01.381559+0000 2024-09-16T05:56:06.022 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 1.0 2024-09-16T05:56:06.377 INFO:tasks.workunit.client.0.smithi178.stderr:instructing pg 1.0 on osd.0 to scrub 2024-09-16T05:56:06.389 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 1.0 2024-09-16T05:56:01.381559+0000 2024-09-16T05:56:06.390 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-09-16T05:56:06.390 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-09-16T05:56:01.381559+0000 2024-09-16T05:56:06.390 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-09-16T05:56:06.390 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-09-16T05:56:06.390 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 1 )) 2024-09-16T05:56:06.390 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T05:56:06.390 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T05:56:06.390 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T05:56:06.390 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T05:56:06.390 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T05:56:06.772 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-09-16T05:56:01.381559+0000 '>' 2024-09-16T05:56:01.381559+0000 2024-09-16T05:56:06.772 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-09-16T05:56:07.774 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-09-16T05:56:07.774 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 1 )) 2024-09-16T05:56:07.774 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: return 1 2024-09-16T05:56:07.774 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1942: test_pg_schedule_scrub: teardown td/ceph-helpers 2024-09-16T05:56:07.774 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:56:07.774 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:56:07.774 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:56:07.774 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:56:07.774 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:56:07.774 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:56:07.774 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:56:07.775 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:56:07.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:56:07.881 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:56:07.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:56:07.883 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:56:07.884 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:56:07.884 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:56:07.884 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:56:07.885 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:56:07.885 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:56:07.885 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:56:07.886 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:56:07.886 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:56:07.887 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:56:07.887 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:56:07.899 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:56:07.899 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:56:07.899 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:56:07.899 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:56:07.900 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:56:07.900 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:56:07.900 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T05:56:07.900 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_pg_scrub td/ceph-helpers 2024-09-16T05:56:07.900 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: test_pg_scrub: local dir=td/ceph-helpers 2024-09-16T05:56:07.900 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: test_pg_scrub: setup td/ceph-helpers 2024-09-16T05:56:07.901 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T05:56:07.901 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T05:56:07.901 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:56:07.901 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:56:07.901 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:56:07.902 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:56:07.902 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:56:07.902 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:56:07.902 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:56:07.902 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:56:07.902 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:56:07.903 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:56:07.906 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:56:07.906 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:56:07.906 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:56:07.907 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:56:07.907 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:56:07.907 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:56:07.907 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:56:07.907 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:56:07.907 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:56:07.908 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:56:07.908 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:56:07.908 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:56:07.908 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:56:07.908 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:56:07.908 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:56:07.909 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:56:07.912 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:56:07.912 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:56:07.913 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T05:56:07.913 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:56:07.913 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:56:07.913 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:56:07.913 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T05:56:07.913 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:56:07.914 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:56:07.914 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:56:07.914 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T05:56:07.914 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: test_pg_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:56:07.914 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T05:56:07.914 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:56:07.915 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:56:07.915 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:56:07.915 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T05:56:07.915 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:56:08.302 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:56:08.302 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:56:08.302 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:56:08.302 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:56:08.302 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:56:08.302 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:56:08.302 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:56:08.303 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:56:08.520 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:56:08.521 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:56:08.521 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:56:08.521 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:56:08.521 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:56:08.522 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:56:08.522 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:56:08.522 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:56:08.522 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:56:08.522 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:56:08.522 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:56:08.523 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:56:08.523 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:56:08.523 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:56:08.523 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T05:56:08.611 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:56:08.611 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:56:08.611 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:56:08.611 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:56:08.611 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:56:08.611 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:56:08.611 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:56:08.612 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:56:08.612 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:56:08.612 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:56:08.612 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:56:08.612 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:56:08.612 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:56:08.612 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T05:56:08.691 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1893: test_pg_scrub: run_mgr td/ceph-helpers x 2024-09-16T05:56:08.691 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T05:56:08.691 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T05:56:08.691 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T05:56:08.692 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T05:56:08.692 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T05:56:08.692 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T05:56:09.540 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T05:56:09.540 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:56:09.541 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:56:09.541 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:56:09.541 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:56:09.541 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:56:09.541 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:56:09.541 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:56:09.541 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:56:09.571 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1894: test_pg_scrub: run_osd td/ceph-helpers 0 2024-09-16T05:56:09.572 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T05:56:09.572 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:56:09.572 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T05:56:09.572 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:56:09.572 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T05:56:09.572 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T05:56:09.572 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:56:09.572 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:56:09.572 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:56:09.572 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T05:56:09.572 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T05:56:09.572 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:56:09.572 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:56:09.573 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T05:56:09.573 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:56:09.573 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:56:09.573 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:56:09.573 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:56:09.573 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:56:09.573 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:56:09.573 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:56:09.573 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:56:09.573 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:56:09.573 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:56:09.573 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:56:09.573 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T05:56:09.573 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T05:56:09.574 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:56:09.574 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:56:09.574 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:56:09.574 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:56:09.574 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:56:09.574 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:56:09.574 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T05:56:09.574 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:56:09.575 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=668d367a-3c7b-4274-af24-ac8c0347ac9f 2024-09-16T05:56:09.575 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 668d367a-3c7b-4274-af24-ac8c0347ac9f' 2024-09-16T05:56:09.575 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 668d367a-3c7b-4274-af24-ac8c0347ac9f 2024-09-16T05:56:09.575 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:56:09.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB5yOdmXNI6IxAAsKrfPuFc3scVZcVdiHe5Bw== 2024-09-16T05:56:09.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB5yOdmXNI6IxAAsKrfPuFc3scVZcVdiHe5Bw=="}' 2024-09-16T05:56:09.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 668d367a-3c7b-4274-af24-ac8c0347ac9f -i td/ceph-helpers/0/new.json 2024-09-16T05:56:09.860 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:56:09.870 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T05:56:09.871 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB5yOdmXNI6IxAAsKrfPuFc3scVZcVdiHe5Bw== --osd-uuid 668d367a-3c7b-4274-af24-ac8c0347ac9f 2024-09-16T05:56:09.900 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:56:09.899+0000 7f87b581a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:56:09.921 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:56:09.920+0000 7f87b581a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:56:09.922 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:56:09.922+0000 7f87b581a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:56:09.923 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:56:09.922+0000 7f87b581a640 -1 bdev(0x5573d6dbbc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-16T05:56:09.923 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:56:09.922+0000 7f87b581a640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T05:56:15.633 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T05:56:15.633 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:56:15.634 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T05:56:15.634 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T05:56:15.635 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:56:16.152 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:56:16.153 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T05:56:16.153 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:56:16.153 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:56:16.153 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:56:16.153 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:56:16.186 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:56:16.178+0000 7f279b0b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:56:16.216 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:56:16.199+0000 7f279b0b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:56:16.216 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:56:16.201+0000 7f279b0b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:56:16.569 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T05:56:16.570 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:56:16.570 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:56:16.570 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:56:16.570 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:56:16.570 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:56:16.570 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:56:16.570 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:56:16.571 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:56:16.571 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:56:16.930 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:56:17.302 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:56:17.302+0000 7f279b0b6640 -1 Falling back to public interface 2024-09-16T05:56:17.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:56:17.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:56:17.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:56:17.934 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:56:17.934 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:56:17.934 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:56:18.312 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:56:19.313 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:56:19.313 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:56:19.314 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:56:19.314 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:56:19.314 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:56:19.314 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:56:19.687 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:56:20.111 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:56:20.110+0000 7f279b0b6640 -1 osd.0 0 log_to_monitors true 2024-09-16T05:56:20.688 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:56:20.688 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:56:20.688 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:56:20.689 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:56:20.689 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:56:20.689 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:56:21.095 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:56:22.091 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:56:22.091 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:56:22.092 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:56:22.092 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:56:22.092 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:56:22.092 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:56:22.474 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:56:23.475 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:56:23.475 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:56:23.476 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:56:23.476 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:56:23.476 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:56:23.476 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:56:23.901 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:56:24.902 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:56:24.903 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:56:24.903 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:56:24.903 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:56:24.904 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:56:24.904 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:56:25.305 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:56:26.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:56:26.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:56:26.307 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:56:26.308 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:56:26.308 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:56:26.308 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:56:26.710 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:56:27.712 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:56:27.712 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:56:27.712 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:56:27.712 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:56:27.713 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:56:27.713 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:56:28.216 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:56:29.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:56:29.218 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:56:29.218 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:56:29.218 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:56:29.218 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:56:29.219 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:56:29.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:56:30.622 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:56:30.623 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:56:30.623 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:56:30.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:56:30.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:56:30.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:56:31.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:56:32.046 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:56:32.046 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:56:32.046 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:56:32.047 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:56:32.047 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:56:32.047 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:56:32.451 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:56:33.452 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:56:33.453 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:56:33.453 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:56:33.453 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:56:33.453 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:56:33.453 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:56:34.681 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:56:35.682 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:56:35.683 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:56:35.683 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:56:35.683 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:56:35.683 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:56:35.683 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:56:36.155 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:56:37.157 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:56:37.157 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:56:37.157 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:56:37.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:56:37.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:56:37.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:56:37.726 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:56:38.727 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:56:38.728 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:56:38.728 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:56:38.728 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:56:38.728 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:56:38.729 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:56:39.435 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:56:40.436 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:56:40.437 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:56:40.437 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:56:40.438 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:56:40.438 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:56:40.438 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:56:41.321 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:56:42.322 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:56:42.322 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:56:42.322 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:56:42.323 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:56:42.323 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:56:42.323 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:56:43.250 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:56:44.251 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:56:44.251 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:56:44.251 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T05:56:44.252 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T05:56:44.252 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:56:44.252 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:56:44.638 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1795853847,v1:127.0.0.1:6803/1795853847] [v2:127.0.0.1:6804/1795853847,v1:127.0.0.1:6805/1795853847] exists,up 668d367a-3c7b-4274-af24-ac8c0347ac9f 2024-09-16T05:56:44.639 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:56:44.639 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:56:44.639 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:56:44.639 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1895: test_pg_scrub: create_rbd_pool 2024-09-16T05:56:44.639 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T05:56:45.037 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' does not exist 2024-09-16T05:56:45.047 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T05:56:45.048 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T05:56:45.525 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' already exists 2024-09-16T05:56:45.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T05:56:46.537 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T05:56:47.566 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1896: test_pg_scrub: wait_for_clean 2024-09-16T05:56:47.567 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T05:56:47.567 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T05:56:47.567 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T05:56:47.567 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T05:56:47.567 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T05:56:47.568 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T05:56:47.568 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T05:56:47.568 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T05:56:47.569 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T05:56:47.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T05:56:47.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T05:56:47.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T05:56:47.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T05:56:47.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-16T05:56:47.662 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-16T05:56:48.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-09-16T05:56:48.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-16T05:56:48.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:56:48.039 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T05:56:48.154 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836484 2024-09-16T05:56:48.154 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836484 2024-09-16T05:56:48.154 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836484' 2024-09-16T05:56:48.154 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:56:48.154 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836484 2024-09-16T05:56:48.154 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:56:48.155 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-16T05:56:48.155 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836484 2024-09-16T05:56:48.155 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:56:48.155 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836484 2024-09-16T05:56:48.155 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.0 seq 21474836484 2024-09-16T05:56:48.156 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2024-09-16T05:56:48.156 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:56:48.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836482 -lt 21474836484 2024-09-16T05:56:48.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-16T05:56:49.531 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T05:56:49.531 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:56:49.953 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836484 -lt 21474836484 2024-09-16T05:56:49.954 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T05:56:49.954 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:56:49.954 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:56:50.454 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T05:56:50.454 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T05:56:50.454 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T05:56:50.454 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T05:56:50.454 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T05:56:50.454 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T05:56:50.454 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T05:56:50.454 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T05:56:50.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T05:56:50.844 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T05:56:50.844 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:56:50.844 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:56:51.342 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T05:56:51.342 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T05:56:51.343 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T05:56:51.343 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1897: test_pg_scrub: pg_scrub 1.0 2024-09-16T05:56:51.343 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=1.0 2024-09-16T05:56:51.343 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 1.0 2024-09-16T05:56:51.343 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T05:56:51.343 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T05:56:51.343 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T05:56:51.343 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T05:56:51.762 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-09-16T05:56:45.360929+0000 2024-09-16T05:56:51.762 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 1.0 2024-09-16T05:56:52.110 INFO:tasks.workunit.client.0.smithi178.stderr:instructing pg 1.0 on osd.0 to scrub 2024-09-16T05:56:52.121 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 1.0 2024-09-16T05:56:45.360929+0000 2024-09-16T05:56:52.121 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-09-16T05:56:52.121 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-09-16T05:56:45.360929+0000 2024-09-16T05:56:52.122 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-09-16T05:56:52.122 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-09-16T05:56:52.122 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-09-16T05:56:52.122 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T05:56:52.122 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T05:56:52.122 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T05:56:52.122 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T05:56:52.122 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T05:56:52.480 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-09-16T05:56:45.360929+0000 '>' 2024-09-16T05:56:45.360929+0000 2024-09-16T05:56:52.480 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-09-16T05:56:53.481 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-09-16T05:56:53.481 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-09-16T05:56:53.482 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T05:56:53.482 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T05:56:53.482 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T05:56:53.482 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T05:56:53.482 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T05:56:53.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-09-16T05:56:45.360929+0000 '>' 2024-09-16T05:56:45.360929+0000 2024-09-16T05:56:53.859 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-09-16T05:56:54.860 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-09-16T05:56:54.860 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-09-16T05:56:54.861 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T05:56:54.861 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T05:56:54.861 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T05:56:54.861 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T05:56:54.861 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T05:56:55.229 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-09-16T05:56:45.360929+0000 '>' 2024-09-16T05:56:45.360929+0000 2024-09-16T05:56:55.229 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-09-16T05:56:56.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-09-16T05:56:56.231 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-09-16T05:56:56.231 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T05:56:56.232 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T05:56:56.232 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T05:56:56.232 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T05:56:56.232 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T05:56:56.609 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-09-16T05:56:52.796309+0000 '>' 2024-09-16T05:56:45.360929+0000 2024-09-16T05:56:56.610 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-09-16T05:56:56.610 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1898: test_pg_scrub: kill_daemons td/ceph-helpers KILL osd 2024-09-16T05:56:56.610 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:56:56.610 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:56:56.610 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:56:56.610 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:56:56.610 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:56:56.715 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:56:56.715 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1899: test_pg_scrub: TIMEOUT=1 2024-09-16T05:56:56.715 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1899: test_pg_scrub: pg_scrub 1.0 2024-09-16T05:56:56.715 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=1.0 2024-09-16T05:56:56.715 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 1.0 2024-09-16T05:56:56.715 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T05:56:56.715 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T05:56:56.715 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T05:56:56.716 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T05:56:57.075 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-09-16T05:56:52.796309+0000 2024-09-16T05:56:57.075 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 1.0 2024-09-16T05:56:57.430 INFO:tasks.workunit.client.0.smithi178.stderr:instructing pg 1.0 on osd.0 to scrub 2024-09-16T05:56:57.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 1.0 2024-09-16T05:56:52.796309+0000 2024-09-16T05:56:57.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-09-16T05:56:57.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-09-16T05:56:52.796309+0000 2024-09-16T05:56:57.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-09-16T05:56:57.442 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-09-16T05:56:57.442 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 1 )) 2024-09-16T05:56:57.442 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T05:56:57.442 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T05:56:57.442 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T05:56:57.442 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T05:56:57.442 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T05:56:57.806 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-09-16T05:56:52.796309+0000 '>' 2024-09-16T05:56:52.796309+0000 2024-09-16T05:56:57.807 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-09-16T05:56:58.809 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-09-16T05:56:58.810 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 1 )) 2024-09-16T05:56:58.810 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: return 1 2024-09-16T05:56:58.810 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1900: test_pg_scrub: teardown td/ceph-helpers 2024-09-16T05:56:58.810 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:56:58.810 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:56:58.810 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:56:58.810 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:56:58.810 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:56:58.810 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:56:58.810 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:56:58.810 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:56:58.916 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:56:58.916 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:56:58.917 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:56:58.917 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:56:58.918 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:56:58.918 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:56:58.919 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:56:58.920 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:56:58.920 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:56:58.920 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:56:58.920 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:56:58.921 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:56:58.922 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:56:58.922 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:56:58.929 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:56:58.929 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:56:58.929 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:56:58.929 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:56:58.931 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:56:58.932 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:56:58.932 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T05:56:58.932 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_repair td/ceph-helpers 2024-09-16T05:56:58.932 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: test_repair: local dir=td/ceph-helpers 2024-09-16T05:56:58.932 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: test_repair: setup td/ceph-helpers 2024-09-16T05:56:58.932 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T05:56:58.932 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T05:56:58.932 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:56:58.932 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:56:58.932 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:56:58.932 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:56:58.932 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:56:58.932 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:56:58.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:56:58.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:56:58.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:56:58.933 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:56:58.934 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:56:58.934 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:56:58.936 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:56:58.936 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:56:58.936 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:56:58.937 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:56:58.938 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:56:58.938 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:56:58.938 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:56:58.938 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:56:58.939 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:56:58.939 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:56:58.941 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:56:58.941 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:56:58.941 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:56:58.941 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:56:58.942 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:56:58.942 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:56:58.942 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T05:56:58.943 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:56:58.944 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:56:58.944 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:56:58.944 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T05:56:58.946 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:56:58.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:56:58.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:56:58.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T05:56:58.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: test_repair: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:56:58.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T05:56:58.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:56:58.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:56:58.947 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:56:58.947 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T05:56:58.947 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:56:59.332 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:56:59.332 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:56:59.332 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:56:59.332 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:56:59.333 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:56:59.333 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:56:59.333 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:56:59.333 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:56:59.533 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:56:59.534 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:56:59.534 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:56:59.534 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:56:59.534 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:56:59.534 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:56:59.535 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:56:59.535 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:56:59.535 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:56:59.535 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:56:59.535 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:56:59.535 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:56:59.535 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:56:59.535 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:56:59.536 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T05:56:59.619 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:56:59.619 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:56:59.619 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:56:59.619 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:56:59.619 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:56:59.619 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:56:59.619 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:56:59.620 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:56:59.620 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:56:59.620 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:56:59.620 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:56:59.620 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:56:59.620 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:56:59.620 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T05:56:59.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: test_repair: run_mgr td/ceph-helpers x 2024-09-16T05:56:59.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T05:56:59.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T05:56:59.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T05:56:59.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T05:56:59.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T05:56:59.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T05:57:00.553 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T05:57:00.553 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:57:00.553 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:57:00.553 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:57:00.553 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:57:00.553 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:57:00.553 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:57:00.554 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:57:00.554 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:57:00.584 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1851: test_repair: run_osd td/ceph-helpers 0 2024-09-16T05:57:00.585 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T05:57:00.585 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:57:00.585 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T05:57:00.585 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:57:00.585 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T05:57:00.585 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T05:57:00.585 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:57:00.585 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:57:00.585 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:57:00.585 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T05:57:00.585 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T05:57:00.585 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:57:00.586 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:57:00.586 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T05:57:00.586 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:57:00.586 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:57:00.586 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:57:00.586 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:57:00.586 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:57:00.586 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:57:00.586 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:57:00.587 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:57:00.587 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:57:00.587 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:57:00.587 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:57:00.587 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T05:57:00.587 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T05:57:00.587 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:57:00.587 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:57:00.587 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:57:00.587 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:57:00.587 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:57:00.587 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:57:00.588 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T05:57:00.588 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:57:00.588 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1728fdce-ef01-4436-a85b-d222de6b4d95 2024-09-16T05:57:00.588 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1728fdce-ef01-4436-a85b-d222de6b4d95' 2024-09-16T05:57:00.589 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 1728fdce-ef01-4436-a85b-d222de6b4d95 2024-09-16T05:57:00.589 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:57:00.604 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCsyOdmIfH7IxAA11MUzeoCMgWuzHyb8uOt4A== 2024-09-16T05:57:00.604 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCsyOdmIfH7IxAA11MUzeoCMgWuzHyb8uOt4A=="}' 2024-09-16T05:57:00.604 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1728fdce-ef01-4436-a85b-d222de6b4d95 -i td/ceph-helpers/0/new.json 2024-09-16T05:57:00.873 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:57:00.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T05:57:00.883 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCsyOdmIfH7IxAA11MUzeoCMgWuzHyb8uOt4A== --osd-uuid 1728fdce-ef01-4436-a85b-d222de6b4d95 2024-09-16T05:57:00.913 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:57:00.912+0000 7f47b4887640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:57:00.942 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:57:00.941+0000 7f47b4887640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:57:00.943 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:57:00.942+0000 7f47b4887640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:57:00.943 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:57:00.943+0000 7f47b4887640 -1 bdev(0x561bc7913c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-16T05:57:00.943 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:57:00.943+0000 7f47b4887640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T05:57:06.385 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T05:57:06.386 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:57:06.386 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T05:57:06.387 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T05:57:06.387 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:57:06.901 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:57:06.901 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T05:57:06.902 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:57:06.902 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:57:06.902 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:57:06.902 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:57:06.928 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:57:06.927+0000 7fa8eab19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:57:06.952 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:57:06.952+0000 7fa8eab19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:57:06.953 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:57:06.953+0000 7fa8eab19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:57:07.349 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T05:57:07.349 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:57:07.350 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:57:07.350 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:57:07.350 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:57:07.350 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:57:07.350 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:57:07.350 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:57:07.350 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:57:07.351 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:57:07.758 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:57:08.563 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:57:08.563+0000 7fa8eab19640 -1 Falling back to public interface 2024-09-16T05:57:08.760 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:57:08.760 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:57:08.760 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:57:08.760 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:57:08.761 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:57:08.761 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:57:09.140 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:57:10.141 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:57:10.141 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:57:10.141 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:57:10.142 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:57:10.142 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:57:10.142 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:57:10.521 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:57:11.280 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:57:11.279+0000 7fa8eab19640 -1 osd.0 0 log_to_monitors true 2024-09-16T05:57:11.522 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:57:11.522 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:57:11.522 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:57:11.523 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:57:11.523 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:57:11.523 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:57:12.005 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:57:13.007 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:57:13.007 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:57:13.007 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:57:13.008 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:57:13.008 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:57:13.008 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:57:13.415 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:57:14.416 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:57:14.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:57:14.417 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:57:14.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:57:14.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:57:14.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:57:14.802 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:57:15.804 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:57:15.804 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:57:15.804 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:57:15.805 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:57:15.805 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:57:15.805 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:57:16.347 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:57:17.348 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:57:17.349 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:57:17.349 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:57:17.349 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:57:17.350 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:57:17.350 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:57:17.750 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:57:18.750 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:57:18.751 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:57:18.751 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:57:18.751 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:57:18.751 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:57:18.752 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:57:19.157 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:57:20.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:57:20.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:57:20.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:57:20.159 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:57:20.160 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:57:20.160 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:57:20.555 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:57:21.556 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:57:21.556 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:57:21.557 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:57:21.557 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:57:21.557 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:57:21.557 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:57:21.994 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:57:22.996 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:57:22.996 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:57:22.997 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:57:22.997 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:57:22.997 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:57:22.998 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:57:23.831 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:57:24.832 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:57:24.832 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:57:24.833 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:57:24.833 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:57:24.833 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:57:24.833 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:57:25.687 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:57:26.688 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:57:26.688 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:57:26.689 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:57:26.689 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:57:26.689 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:57:26.689 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:57:27.508 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:57:28.509 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:57:28.509 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:57:28.509 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:57:28.510 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:57:28.510 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:57:28.510 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:57:29.007 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:57:30.008 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:57:30.008 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:57:30.009 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:57:30.009 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:57:30.009 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:57:30.009 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:57:30.648 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:57:31.649 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:57:31.650 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:57:31.650 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:57:31.650 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:57:31.651 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:57:31.651 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:57:32.499 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:57:33.501 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:57:33.501 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:57:33.501 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:57:33.501 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:57:33.502 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:57:33.502 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:57:33.962 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3852650601,v1:127.0.0.1:6803/3852650601] [v2:127.0.0.1:6804/3852650601,v1:127.0.0.1:6805/3852650601] exists,up 1728fdce-ef01-4436-a85b-d222de6b4d95 2024-09-16T05:57:33.963 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:57:33.963 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:57:33.963 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:57:33.963 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1852: test_repair: create_rbd_pool 2024-09-16T05:57:33.963 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T05:57:34.341 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' does not exist 2024-09-16T05:57:34.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T05:57:34.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T05:57:34.972 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' already exists 2024-09-16T05:57:34.983 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T05:57:35.986 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T05:57:37.060 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1853: test_repair: wait_for_clean 2024-09-16T05:57:37.060 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T05:57:37.060 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T05:57:37.060 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T05:57:37.061 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T05:57:37.061 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T05:57:37.061 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T05:57:37.061 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T05:57:37.061 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T05:57:37.062 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T05:57:37.156 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T05:57:37.157 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T05:57:37.157 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T05:57:37.157 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T05:57:37.157 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-16T05:57:37.157 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-16T05:57:37.568 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-09-16T05:57:37.569 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-16T05:57:37.569 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T05:57:37.569 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T05:57:37.682 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836489 2024-09-16T05:57:37.683 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836489 2024-09-16T05:57:37.683 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489' 2024-09-16T05:57:37.683 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T05:57:37.683 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836489 2024-09-16T05:57:37.683 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T05:57:37.684 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-16T05:57:37.684 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836489 2024-09-16T05:57:37.684 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T05:57:37.685 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836489 2024-09-16T05:57:37.685 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-09-16T05:57:37.686 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.0 seq 21474836489 2024-09-16T05:57:37.686 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:57:38.054 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836487 -lt 21474836489 2024-09-16T05:57:38.054 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-16T05:57:39.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T05:57:39.055 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T05:57:39.432 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836489 -lt 21474836489 2024-09-16T05:57:39.432 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T05:57:39.432 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:57:39.433 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:57:39.958 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T05:57:39.958 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T05:57:39.958 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T05:57:39.958 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T05:57:39.958 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T05:57:39.958 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T05:57:39.959 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T05:57:39.959 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T05:57:40.337 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T05:57:40.337 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T05:57:40.337 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T05:57:40.337 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T05:57:40.844 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T05:57:40.844 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T05:57:40.844 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T05:57:40.845 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1854: test_repair: repair 1.0 2024-09-16T05:57:40.845 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-09-16T05:57:40.845 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-09-16T05:57:40.845 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T05:57:40.845 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T05:57:40.845 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T05:57:40.845 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T05:57:41.209 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-09-16T05:57:34.762070+0000 2024-09-16T05:57:41.209 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-09-16T05:57:41.562 INFO:tasks.workunit.client.0.smithi178.stderr:instructing pg 1.0 on osd.0 to repair 2024-09-16T05:57:41.574 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-09-16T05:57:34.762070+0000 2024-09-16T05:57:41.574 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-09-16T05:57:41.574 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-09-16T05:57:34.762070+0000 2024-09-16T05:57:41.575 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-09-16T05:57:41.575 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-09-16T05:57:41.575 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-09-16T05:57:41.575 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T05:57:41.575 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T05:57:41.575 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T05:57:41.575 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T05:57:41.575 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T05:57:41.945 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-09-16T05:57:34.762070+0000 '>' 2024-09-16T05:57:34.762070+0000 2024-09-16T05:57:41.947 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-09-16T05:57:42.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-09-16T05:57:42.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-09-16T05:57:42.947 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T05:57:42.947 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T05:57:42.947 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T05:57:42.947 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T05:57:42.947 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T05:57:43.319 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-09-16T05:57:34.762070+0000 '>' 2024-09-16T05:57:34.762070+0000 2024-09-16T05:57:43.319 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-09-16T05:57:44.320 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-09-16T05:57:44.320 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-09-16T05:57:44.321 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T05:57:44.321 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T05:57:44.321 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T05:57:44.321 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T05:57:44.321 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T05:57:44.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-09-16T05:57:34.762070+0000 '>' 2024-09-16T05:57:34.762070+0000 2024-09-16T05:57:44.699 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-09-16T05:57:45.699 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-09-16T05:57:45.699 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-09-16T05:57:45.700 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T05:57:45.700 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T05:57:45.700 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T05:57:45.700 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T05:57:45.700 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T05:57:46.077 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-09-16T05:57:34.762070+0000 '>' 2024-09-16T05:57:34.762070+0000 2024-09-16T05:57:46.077 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-09-16T05:57:47.078 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-09-16T05:57:47.078 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-09-16T05:57:47.079 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T05:57:47.079 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T05:57:47.079 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T05:57:47.079 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T05:57:47.079 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T05:57:47.451 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-09-16T05:57:42.461385+0000 '>' 2024-09-16T05:57:34.762070+0000 2024-09-16T05:57:47.451 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-09-16T05:57:47.452 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: test_repair: kill_daemons td/ceph-helpers KILL osd 2024-09-16T05:57:47.452 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:57:47.452 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:57:47.452 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:57:47.452 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:57:47.452 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:57:47.557 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:57:47.557 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: test_repair: TIMEOUT=1 2024-09-16T05:57:47.557 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: test_repair: repair 1.0 2024-09-16T05:57:47.557 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-09-16T05:57:47.557 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-09-16T05:57:47.557 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T05:57:47.558 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T05:57:47.558 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T05:57:47.558 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T05:57:47.921 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-09-16T05:57:42.461385+0000 2024-09-16T05:57:47.921 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-09-16T05:57:48.274 INFO:tasks.workunit.client.0.smithi178.stderr:instructing pg 1.0 on osd.0 to repair 2024-09-16T05:57:48.287 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-09-16T05:57:42.461385+0000 2024-09-16T05:57:48.287 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-09-16T05:57:48.287 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-09-16T05:57:42.461385+0000 2024-09-16T05:57:48.287 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-09-16T05:57:48.287 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-09-16T05:57:48.287 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 1 )) 2024-09-16T05:57:48.287 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T05:57:48.287 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T05:57:48.287 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T05:57:48.288 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T05:57:48.288 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T05:57:48.652 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-09-16T05:57:42.461385+0000 '>' 2024-09-16T05:57:42.461385+0000 2024-09-16T05:57:48.652 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-09-16T05:57:49.654 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-09-16T05:57:49.654 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 1 )) 2024-09-16T05:57:49.654 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: return 1 2024-09-16T05:57:49.654 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: test_repair: teardown td/ceph-helpers 2024-09-16T05:57:49.654 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:57:49.654 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:57:49.655 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:57:49.655 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:57:49.655 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:57:49.655 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:57:49.655 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:57:49.655 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:57:49.761 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:57:49.762 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:57:49.762 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:57:49.763 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:57:49.764 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:57:49.764 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:57:49.764 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:57:49.765 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:57:49.765 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:57:49.766 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:57:49.766 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:57:49.766 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:57:49.768 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:57:49.768 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:57:49.775 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:57:49.775 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:57:49.775 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:57:49.775 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:57:49.777 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:57:49.778 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:57:49.778 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T05:57:49.778 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_run_in_background td/ceph-helpers 2024-09-16T05:57:49.778 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: test_run_in_background: local pids 2024-09-16T05:57:49.778 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: test_run_in_background: run_in_background pids sleep 1 2024-09-16T05:57:49.778 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-09-16T05:57:49.778 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-09-16T05:57:49.778 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 138820"' 2024-09-16T05:57:49.778 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 138820' 2024-09-16T05:57:49.778 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: test_run_in_background: run_in_background pids sleep 1 2024-09-16T05:57:49.778 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-09-16T05:57:49.779 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-09-16T05:57:49.779 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sleep 1 2024-09-16T05:57:49.779 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 138822"' 2024-09-16T05:57:49.779 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 138822' 2024-09-16T05:57:49.779 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/138823: /' 2024-09-16T05:57:49.779 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sleep 1 2024-09-16T05:57:49.779 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/138826: /' 2024-09-16T05:57:49.779 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: test_run_in_background: echo 138820 138822 2024-09-16T05:57:49.779 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: test_run_in_background: wc -w 2024-09-16T05:57:49.792 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: test_run_in_background: test 2 = 2 2024-09-16T05:57:49.792 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: test_run_in_background: wait 138820 138822 2024-09-16T05:57:50.779 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-09-16T05:57:50.779 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-09-16T05:57:50.779 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T05:57:50.779 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_run_mon td/ceph-helpers 2024-09-16T05:57:50.779 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:495: test_run_mon: local dir=td/ceph-helpers 2024-09-16T05:57:50.779 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:497: test_run_mon: setup td/ceph-helpers 2024-09-16T05:57:50.779 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T05:57:50.779 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T05:57:50.779 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:57:50.779 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:57:50.780 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:57:50.780 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:57:50.780 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:57:50.780 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:57:50.780 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:57:50.780 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:57:50.781 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:57:50.781 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:57:50.782 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:57:50.782 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:57:50.783 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:57:50.783 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:57:50.784 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:57:50.785 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:57:50.785 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:57:50.785 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:57:50.785 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:57:50.786 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:57:50.787 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:57:50.787 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:57:50.789 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:57:50.789 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:57:50.789 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:57:50.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:57:50.790 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:57:50.790 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:57:50.790 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T05:57:50.791 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:57:50.792 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:57:50.792 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:57:50.792 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T05:57:50.794 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:57:50.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:57:50.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:57:50.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T05:57:50.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:499: test_run_mon: run_mon td/ceph-helpers a 2024-09-16T05:57:50.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T05:57:50.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:57:50.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:57:50.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:57:50.795 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T05:57:50.795 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-16T05:57:51.185 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:57:51.185 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:57:51.185 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:57:51.185 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:57:51.185 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:57:51.185 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:57:51.185 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:57:51.185 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T05:57:51.392 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:57:51.393 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:57:51.393 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:57:51.393 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:57:51.393 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:57:51.395 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:57:51.395 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:57:51.395 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:57:51.395 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:57:51.395 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:57:51.395 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:57:51.395 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:57:51.395 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:57:51.395 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:57:51.395 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T05:57:51.481 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:57:51.481 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:57:51.481 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:57:51.481 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:57:51.481 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:57:51.481 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:57:51.481 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:57:51.481 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:57:51.481 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:57:51.481 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:57:51.482 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:57:51.482 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:57:51.482 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:57:51.482 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T05:57:51.559 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:500: test_run_mon: ceph mon dump 2024-09-16T05:57:51.559 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:500: test_run_mon: grep mon.a 2024-09-16T05:57:52.346 INFO:tasks.workunit.client.0.smithi178.stderr:dumped monmap epoch 1 2024-09-16T05:57:52.355 INFO:tasks.workunit.client.0.smithi178.stdout:0: v2:127.0.0.1:7109/0 mon.a 2024-09-16T05:57:52.355 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:501: test_run_mon: kill_daemons td/ceph-helpers 2024-09-16T05:57:52.356 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:57:52.356 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:57:52.357 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:57:52.357 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:57:52.357 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:57:52.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:57:52.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:503: test_run_mon: run_mon td/ceph-helpers a --osd_pool_default_size=3 2024-09-16T05:57:52.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T05:57:52.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:57:52.663 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:57:52.663 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:57:52.663 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T05:57:52.663 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=3 2024-09-16T05:57:52.684 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:57:52.683+0000 7f5416a14c80 -1 'td/ceph-helpers/a' already exists and is not empty: monitor may already exist 2024-09-16T05:57:52.686 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:57:52.686 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:57:52.686 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:57:52.686 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:57:52.686 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:57:52.686 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:57:52.686 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:57:52.686 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-16T05:57:52.864 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:57:52.865 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:57:52.865 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:57:52.865 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:57:52.865 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:57:52.866 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:57:52.866 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:57:52.866 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:57:52.866 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:57:52.866 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:57:52.866 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:57:52.866 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:57:52.866 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:57:52.866 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:57:52.866 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T05:57:52.944 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:57:52.944 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:57:52.944 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:57:52.944 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:57:52.944 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:57:52.944 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:57:52.945 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:57:52.945 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:57:52.945 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:57:52.945 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:57:52.945 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:57:52.945 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:57:52.945 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:57:52.945 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T05:57:53.024 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:504: test_run_mon: run_osd td/ceph-helpers 0 2024-09-16T05:57:53.024 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T05:57:53.024 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:57:53.024 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T05:57:53.024 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:57:53.024 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T05:57:53.024 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T05:57:53.025 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:57:53.025 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:57:53.025 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:57:53.025 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T05:57:53.025 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T05:57:53.025 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:57:53.025 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:57:53.025 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T05:57:53.025 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:57:53.025 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:57:53.025 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:57:53.025 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:57:53.025 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:57:53.025 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:57:53.026 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:57:53.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:57:53.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:57:53.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:57:53.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:57:53.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T05:57:53.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T05:57:53.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:57:53.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:57:53.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:57:53.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:57:53.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:57:53.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:57:53.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T05:57:53.027 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:57:53.027 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0c703e9f-9530-4d79-8108-ca37f641963a 2024-09-16T05:57:53.027 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 0c703e9f-9530-4d79-8108-ca37f641963a 2024-09-16T05:57:53.028 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0c703e9f-9530-4d79-8108-ca37f641963a' 2024-09-16T05:57:53.028 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:57:53.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDhyOdm8OugAhAAlmeL3Whkwk941RoGTdnQcQ== 2024-09-16T05:57:53.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDhyOdm8OugAhAAlmeL3Whkwk941RoGTdnQcQ=="}' 2024-09-16T05:57:53.045 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0c703e9f-9530-4d79-8108-ca37f641963a -i td/ceph-helpers/0/new.json 2024-09-16T05:57:53.302 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:57:53.311 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T05:57:53.312 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDhyOdm8OugAhAAlmeL3Whkwk941RoGTdnQcQ== --osd-uuid 0c703e9f-9530-4d79-8108-ca37f641963a 2024-09-16T05:57:53.341 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:57:53.340+0000 7f1a9a2e2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:57:53.370 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:57:53.370+0000 7f1a9a2e2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:57:53.371 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:57:53.371+0000 7f1a9a2e2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:57:53.372 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:57:53.372+0000 7f1a9a2e2640 -1 bdev(0x55b2a740bc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-16T05:57:53.372 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:57:53.372+0000 7f1a9a2e2640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T05:57:59.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T05:57:59.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:57:59.308 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T05:57:59.308 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T05:57:59.308 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:57:59.511 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:57:59.512 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T05:57:59.512 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:57:59.512 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:57:59.512 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:57:59.512 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:57:59.541 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:57:59.540+0000 7f78a13ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:57:59.544 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:57:59.544+0000 7f78a13ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:57:59.545 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:57:59.545+0000 7f78a13ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:57:59.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T05:57:59.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:57:59.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:57:59.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:57:59.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:57:59.720 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:57:59.720 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:57:59.720 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:57:59.720 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:57:59.721 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:57:59.934 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:00.881 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:58:00.881+0000 7f78a13ae640 -1 Falling back to public interface 2024-09-16T05:58:00.935 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:00.935 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:00.936 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:58:00.936 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:58:00.936 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:00.936 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:58:01.129 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:02.130 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:02.130 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:02.130 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:58:02.131 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:58:02.131 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:02.131 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:58:02.327 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:03.329 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:03.329 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:03.329 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:58:03.329 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:58:03.330 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:03.330 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:58:03.521 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:03.601 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:58:03.601+0000 7f78a13ae640 -1 osd.0 0 log_to_monitors true 2024-09-16T05:58:04.523 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:04.523 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:04.523 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:58:04.524 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:58:04.524 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:04.524 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:58:04.715 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:05.716 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:05.717 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:05.717 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:58:05.717 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:58:05.717 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:05.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:58:05.912 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:06.914 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:06.914 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:06.915 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:58:06.915 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:58:06.915 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:06.915 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:58:07.109 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:08.110 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:08.111 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:08.111 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:58:08.111 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:58:08.112 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:08.112 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:58:08.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:09.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:09.308 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:09.308 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:58:09.308 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:58:09.308 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:09.308 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:58:09.502 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:10.503 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:10.503 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:10.503 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:58:10.504 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:58:10.504 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:10.504 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:58:10.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:11.700 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:11.700 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:11.701 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:58:11.701 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:58:11.701 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:11.701 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:58:11.894 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:12.895 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:12.896 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:58:12.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:12.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:58:12.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:12.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:58:13.092 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:14.094 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:14.094 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:14.094 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:58:14.094 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:58:14.095 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:14.095 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:58:14.291 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:15.292 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:15.293 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:15.293 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:58:15.293 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:58:15.293 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:15.294 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:58:15.488 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:16.489 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:16.490 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:16.490 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:58:16.490 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:58:16.490 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:16.491 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:58:16.683 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:17.684 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:17.684 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:17.685 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:58:17.685 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:58:17.685 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:17.686 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:58:17.872 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:18.873 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:18.873 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:18.873 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:58:18.874 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:58:18.874 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:18.874 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:58:19.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:20.059 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:20.059 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:20.059 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:58:20.060 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:58:20.060 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:20.060 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:58:20.250 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:21.251 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:21.251 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:21.252 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T05:58:21.252 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T05:58:21.252 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:21.252 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:58:21.442 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:22.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:22.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:22.444 INFO:tasks.workunit.client.0.smithi178.stdout:19 2024-09-16T05:58:22.444 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-16T05:58:22.444 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:22.444 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:58:22.638 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:23.640 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:23.640 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:23.640 INFO:tasks.workunit.client.0.smithi178.stdout:20 2024-09-16T05:58:23.641 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-16T05:58:23.641 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:23.641 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:58:23.836 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:24.837 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:24.838 INFO:tasks.workunit.client.0.smithi178.stdout:21 2024-09-16T05:58:24.838 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:24.838 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-09-16T05:58:24.838 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:24.838 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:58:25.033 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:26.035 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:26.035 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:26.035 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-09-16T05:58:26.035 INFO:tasks.workunit.client.0.smithi178.stdout:22 2024-09-16T05:58:26.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:26.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:58:26.225 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2200711279,v1:127.0.0.1:6801/2200711279] [v2:127.0.0.1:6802/2200711279,v1:127.0.0.1:6803/2200711279] exists,up 0c703e9f-9530-4d79-8108-ca37f641963a 2024-09-16T05:58:26.229 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:58:26.229 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:58:26.229 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:58:26.229 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:505: test_run_mon: run_osd td/ceph-helpers 1 2024-09-16T05:58:26.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T05:58:26.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:58:26.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T05:58:26.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:58:26.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-09-16T05:58:26.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T05:58:26.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:58:26.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:58:26.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:58:26.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-16T05:58:26.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-16T05:58:26.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:58:26.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:58:26.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T05:58:26.231 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:58:26.231 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:58:26.231 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:58:26.231 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:58:26.231 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:58:26.231 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:58:26.231 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:58:26.231 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:58:26.231 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:58:26.231 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:58:26.231 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:58:26.231 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T05:58:26.231 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T05:58:26.231 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:58:26.232 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:58:26.232 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:58:26.232 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:58:26.232 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:58:26.232 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:58:26.232 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-09-16T05:58:26.232 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:58:26.232 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=41d327cb-139a-4657-9f1b-427db1fcffa5 2024-09-16T05:58:26.232 INFO:tasks.workunit.client.0.smithi178.stdout:add osd1 41d327cb-139a-4657-9f1b-427db1fcffa5 2024-09-16T05:58:26.233 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 41d327cb-139a-4657-9f1b-427db1fcffa5' 2024-09-16T05:58:26.233 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:58:26.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQACyedmNcPDDhAA7lCqH120LcUdggEe314OVA== 2024-09-16T05:58:26.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQACyedmNcPDDhAA7lCqH120LcUdggEe314OVA=="}' 2024-09-16T05:58:26.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 41d327cb-139a-4657-9f1b-427db1fcffa5 -i td/ceph-helpers/1/new.json 2024-09-16T05:58:26.533 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:58:26.542 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-09-16T05:58:26.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQACyedmNcPDDhAA7lCqH120LcUdggEe314OVA== --osd-uuid 41d327cb-139a-4657-9f1b-427db1fcffa5 2024-09-16T05:58:26.572 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:58:26.571+0000 7f8ac3bb2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:58:26.573 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:58:26.573+0000 7f8ac3bb2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:58:26.574 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:58:26.574+0000 7f8ac3bb2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:58:26.575 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:58:26.574+0000 7f8ac3bb2640 -1 bdev(0x55eac739dc00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2024-09-16T05:58:26.575 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:58:26.574+0000 7f8ac3bb2640 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-16T05:58:31.975 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-16T05:58:31.975 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:58:31.976 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd1 key to auth repository 2024-09-16T05:58:31.976 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T05:58:31.976 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:58:32.179 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T05:58:32.180 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.1 2024-09-16T05:58:32.180 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:58:32.181 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:58:32.181 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:58:32.181 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:58:32.206 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:58:32.205+0000 7f03dcc17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:58:32.220 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:58:32.219+0000 7f03dcc17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:58:32.221 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:58:32.220+0000 7f03dcc17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:58:32.393 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T05:58:32.393 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:58:32.393 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T05:58:32.394 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:58:32.394 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:58:32.394 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:32.394 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:58:32.394 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:58:32.394 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:32.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:58:32.603 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:33.555 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:58:33.555+0000 7f03dcc17640 -1 Falling back to public interface 2024-09-16T05:58:33.604 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:33.604 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:33.604 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:58:33.605 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:58:33.605 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:33.605 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:58:33.797 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:34.798 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:34.798 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:34.799 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:58:34.799 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:58:34.799 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:34.799 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:58:34.993 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:35.994 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:35.994 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:35.994 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:58:35.995 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:58:35.995 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:35.995 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:58:36.181 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:36.561 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:58:36.561+0000 7f03dcc17640 -1 osd.1 0 log_to_monitors true 2024-09-16T05:58:37.182 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:37.182 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:37.182 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:58:37.183 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:58:37.183 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:37.183 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:58:37.414 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:38.416 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:38.416 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:38.417 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:58:38.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:58:38.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:38.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:58:38.611 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:39.612 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:39.612 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:39.613 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:58:39.613 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:58:39.613 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:39.613 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:58:39.807 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:40.808 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:40.809 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:40.809 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:58:40.809 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:58:40.809 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:40.810 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:58:41.004 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:42.005 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:42.006 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:58:42.006 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:42.006 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:58:42.006 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:42.006 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:58:42.200 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:43.201 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:43.201 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:58:43.202 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:43.202 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:58:43.202 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:43.202 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:58:43.397 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:44.398 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:44.399 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:58:44.399 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:44.399 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:58:44.399 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:44.399 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:58:44.595 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:45.596 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:45.596 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:45.597 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:58:45.597 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:58:45.597 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:45.597 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:58:45.793 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:46.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:46.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:46.794 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:58:46.795 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:58:46.795 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:46.795 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:58:46.988 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:47.989 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:47.989 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:47.989 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:58:47.990 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:58:47.990 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:47.990 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:58:48.183 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:49.184 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:49.184 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:49.185 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:58:49.185 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:58:49.185 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:49.185 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:58:49.379 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:50.380 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:50.381 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:50.381 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:58:50.381 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:58:50.382 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:50.382 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:58:50.572 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:51.573 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:51.573 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:51.573 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:58:51.574 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:58:51.574 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:51.574 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:58:51.769 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:52.770 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:52.770 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:52.770 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:58:52.771 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:58:52.771 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:52.771 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:58:52.966 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:53.967 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:53.968 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:53.968 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T05:58:53.968 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T05:58:53.969 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:53.969 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:58:54.161 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:55.162 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:55.162 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:55.163 INFO:tasks.workunit.client.0.smithi178.stdout:19 2024-09-16T05:58:55.163 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-16T05:58:55.163 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:55.163 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:58:55.354 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:56.355 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:56.356 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:56.356 INFO:tasks.workunit.client.0.smithi178.stdout:20 2024-09-16T05:58:56.356 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-16T05:58:56.357 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:56.357 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:58:56.552 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:58:57.556 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:58:57.556 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:58:57.556 INFO:tasks.workunit.client.0.smithi178.stdout:21 2024-09-16T05:58:57.557 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-09-16T05:58:57.557 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:58:57.557 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T05:58:57.777 INFO:tasks.workunit.client.0.smithi178.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/3745859784,v1:127.0.0.1:6809/3745859784] [v2:127.0.0.1:6810/3745859784,v1:127.0.0.1:6811/3745859784] exists,up 41d327cb-139a-4657-9f1b-427db1fcffa5 2024-09-16T05:58:57.781 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:58:57.781 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:58:57.781 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:58:57.781 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:506: test_run_mon: run_osd td/ceph-helpers 2 2024-09-16T05:58:57.782 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T05:58:57.782 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:58:57.782 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T05:58:57.782 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:58:57.782 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/2 2024-09-16T05:58:57.782 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T05:58:57.782 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:58:57.782 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:58:57.782 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:58:57.782 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2024-09-16T05:58:57.783 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2024-09-16T05:58:57.783 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:58:57.783 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:58:57.783 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T05:58:57.783 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:58:57.783 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:58:57.783 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:58:57.783 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:58:57.783 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:58:57.784 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:58:57.784 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:58:57.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:58:57.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:58:57.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:58:57.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:58:57.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T05:58:57.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T05:58:57.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:58:57.785 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:58:57.785 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:58:57.785 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:58:57.785 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:58:57.785 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:58:57.785 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/2 2024-09-16T05:58:57.785 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:58:57.785 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=be8e2a1c-4130-4d45-b3a6-fb137d794c6d 2024-09-16T05:58:57.785 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 be8e2a1c-4130-4d45-b3a6-fb137d794c6d' 2024-09-16T05:58:57.785 INFO:tasks.workunit.client.0.smithi178.stdout:add osd2 be8e2a1c-4130-4d45-b3a6-fb137d794c6d 2024-09-16T05:58:57.786 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:58:57.799 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAhyedmMQyeLxAATOq09Ak2ITubg573FOzgjQ== 2024-09-16T05:58:57.799 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAhyedmMQyeLxAATOq09Ak2ITubg573FOzgjQ=="}' 2024-09-16T05:58:57.799 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new be8e2a1c-4130-4d45-b3a6-fb137d794c6d -i td/ceph-helpers/2/new.json 2024-09-16T05:58:58.092 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:58:58.101 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/2/new.json 2024-09-16T05:58:58.102 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAhyedmMQyeLxAATOq09Ak2ITubg573FOzgjQ== --osd-uuid be8e2a1c-4130-4d45-b3a6-fb137d794c6d 2024-09-16T05:58:58.130 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:58:58.130+0000 7f27035db640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:58:58.169 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:58:58.169+0000 7f27035db640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:58:58.170 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:58:58.170+0000 7f27035db640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:58:58.172 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:58:58.170+0000 7f27035db640 -1 bdev(0x55d1b8d21c00 td/ceph-helpers/2/block) open stat got: (1) Operation not permitted 2024-09-16T05:58:58.172 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:58:58.170+0000 7f27035db640 -1 bluestore(td/ceph-helpers/2) _read_fsid unparsable uuid 2024-09-16T05:59:03.558 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/2/keyring 2024-09-16T05:59:03.558 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:59:03.559 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd2 key to auth repository 2024-09-16T05:59:03.559 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T05:59:03.559 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:59:03.762 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.2 2024-09-16T05:59:03.763 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T05:59:03.763 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:59:03.763 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:59:03.763 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:59:03.764 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:59:03.790 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:59:03.789+0000 7fab73613640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:59:03.795 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:59:03.795+0000 7fab73613640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:59:03.796 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:59:03.796+0000 7fab73613640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:59:03.968 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T05:59:03.969 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:59:03.969 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T05:59:03.969 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:59:03.969 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:59:03.969 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:59:03.969 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:59:03.970 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:59:03.970 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:59:03.970 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:59:04.178 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:59:04.913 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:59:04.913+0000 7fab73613640 -1 Falling back to public interface 2024-09-16T05:59:05.179 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:59:05.180 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:59:05.180 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:59:05.180 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:59:05.180 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:59:05.180 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:59:05.363 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:59:06.365 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:59:06.365 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:59:06.366 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:59:06.366 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:59:06.366 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:59:06.367 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:59:06.561 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:59:07.562 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:59:07.562 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:59:07.563 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:59:07.563 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:59:07.563 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:59:07.563 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:59:07.563 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:59:07.563+0000 7fab73613640 -1 osd.2 0 log_to_monitors true 2024-09-16T05:59:07.803 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:59:08.805 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:59:08.805 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:59:08.806 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:59:08.806 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:59:08.807 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:59:08.807 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:59:09.002 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:59:10.003 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:59:10.003 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:59:10.004 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:59:10.004 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:59:10.004 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:59:10.004 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:59:10.197 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:59:11.198 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:59:11.198 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:59:11.199 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:59:11.199 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:59:11.199 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:59:11.199 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:59:11.391 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:59:12.393 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:59:12.393 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:59:12.393 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:59:12.394 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:59:12.394 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:59:12.394 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:59:12.589 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:59:13.590 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:59:13.590 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:59:13.591 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:59:13.591 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:59:13.591 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:59:13.591 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:59:13.786 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:59:14.787 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:59:14.787 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:59:14.787 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T05:59:14.788 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T05:59:14.788 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:59:14.788 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:59:14.978 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:59:15.980 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:59:15.980 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:59:15.980 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T05:59:15.981 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T05:59:15.981 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:59:15.981 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:59:16.176 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:59:17.178 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:59:17.178 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:59:17.178 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T05:59:17.179 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T05:59:17.179 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:59:17.179 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:59:17.374 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:59:18.375 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:59:18.376 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:59:18.376 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T05:59:18.376 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T05:59:18.376 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:59:18.376 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:59:18.565 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:59:19.567 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:59:19.567 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:59:19.567 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T05:59:19.568 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T05:59:19.568 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:59:19.568 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:59:19.761 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:59:20.762 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:59:20.762 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:59:20.762 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T05:59:20.763 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T05:59:20.763 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:59:20.763 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:59:20.956 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:59:21.957 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:59:21.957 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:59:21.957 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T05:59:21.958 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T05:59:21.958 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:59:21.958 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:59:22.153 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:59:23.154 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:59:23.155 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:59:23.155 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T05:59:23.155 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T05:59:23.155 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:59:23.155 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:59:23.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:59:24.354 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:59:24.354 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:59:24.355 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T05:59:24.355 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T05:59:24.355 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:59:24.355 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:59:24.547 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:59:25.548 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:59:25.549 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:59:25.549 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T05:59:25.549 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T05:59:25.549 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:59:25.550 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:59:25.747 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:59:26.748 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:59:26.749 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:59:26.749 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-16T05:59:26.749 INFO:tasks.workunit.client.0.smithi178.stdout:19 2024-09-16T05:59:26.749 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:59:26.750 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:59:26.942 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:59:27.943 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:59:27.944 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:59:27.944 INFO:tasks.workunit.client.0.smithi178.stdout:20 2024-09-16T05:59:27.945 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-16T05:59:27.945 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:59:27.945 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:59:28.139 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:59:29.140 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:59:29.141 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:59:29.141 INFO:tasks.workunit.client.0.smithi178.stdout:21 2024-09-16T05:59:29.141 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-09-16T05:59:29.142 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:59:29.142 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T05:59:29.331 INFO:tasks.workunit.client.0.smithi178.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/2462638317,v1:127.0.0.1:6817/2462638317] [v2:127.0.0.1:6818/2462638317,v1:127.0.0.1:6819/2462638317] exists,up be8e2a1c-4130-4d45-b3a6-fb137d794c6d 2024-09-16T05:59:29.332 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T05:59:29.332 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T05:59:29.332 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T05:59:29.332 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:507: test_run_mon: create_rbd_pool 2024-09-16T05:59:29.332 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T05:59:29.548 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' does not exist 2024-09-16T05:59:29.557 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T05:59:29.558 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T05:59:29.982 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' already exists 2024-09-16T05:59:29.992 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T05:59:30.993 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T05:59:33.267 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:508: test_run_mon: ceph osd dump 2024-09-16T05:59:33.267 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:508: test_run_mon: grep 'pool 1 '\''rbd'\''' 2024-09-16T05:59:33.458 INFO:tasks.workunit.client.0.smithi178.stdout:pool 1 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 4 pgp_num 4 autoscale_mode off last_change 24 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.50 2024-09-16T05:59:33.459 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: get_asok_path mon.a 2024-09-16T05:59:33.459 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:59:33.459 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:59:33.459 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:59:33.459 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:59:33.459 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:59:33.459 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:59:33.460 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: CEPH_ARGS= 2024-09-16T05:59:33.460 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: ceph --format=json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get osd_pool_default_size 2024-09-16T05:59:33.535 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: local 'size={"osd_pool_default_size":"3"}' 2024-09-16T05:59:33.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:511: test_run_mon: test '{"osd_pool_default_size":"3"}' = '{"osd_pool_default_size":"3"}' 2024-09-16T05:59:33.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:513: test_run_mon: CEPH_ARGS= 2024-09-16T05:59:33.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:513: test_run_mon: ceph status 2024-09-16T05:59:33.605 INFO:tasks.workunit.client.0.smithi178.stderr:unable to get monitor info from DNS SRV with service name: ceph-mon 2024-09-16T05:59:33.605 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:59:33.604+0000 7f357a9c4640 -1 failed for service _ceph-mon._tcp 2024-09-16T05:59:33.605 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:59:33.604+0000 7f357a9c4640 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-09-16T05:59:33.605 INFO:tasks.workunit.client.0.smithi178.stderr:[errno 2] RADOS object not found (error connecting to the cluster) 2024-09-16T05:59:33.606 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:514: test_run_mon: CEPH_ARGS= 2024-09-16T05:59:33.606 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:514: test_run_mon: ceph --conf td/ceph-helpers/ceph.conf status 2024-09-16T05:59:33.676 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:59:33.675+0000 7f08898fa640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-09-16T05:59:33.676 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:59:33.675+0000 7f08898fa640 -1 AuthRegistry(0x7f0884068d10) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-09-16T05:59:33.676 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:59:33.675+0000 7f08898fa640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-09-16T05:59:33.676 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:59:33.675+0000 7f08898fa640 -1 AuthRegistry(0x7f08898f8fe0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-09-16T05:59:33.677 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:59:33.677+0000 7f08898fa640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-09-16T05:59:33.677 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:59:33.677+0000 7f08898fa640 -1 AuthRegistry(0x7f08840662e0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-09-16T05:59:33.677 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:59:33.677+0000 7f08898fa640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-09-16T05:59:33.677 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:59:33.677+0000 7f08898fa640 -1 AuthRegistry(0x7f0884102290) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-09-16T05:59:33.678 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:59:33.677+0000 7f08898fa640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-09-16T05:59:33.678 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:59:33.677+0000 7f08898fa640 -1 AuthRegistry(0x7f08840662e0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-09-16T05:59:33.777 INFO:tasks.workunit.client.0.smithi178.stdout: cluster: 2024-09-16T05:59:33.777 INFO:tasks.workunit.client.0.smithi178.stdout: id: e36fe3b7-e285-4199-9297-eafea05d044b 2024-09-16T05:59:33.778 INFO:tasks.workunit.client.0.smithi178.stdout: health: HEALTH_OK 2024-09-16T05:59:33.778 INFO:tasks.workunit.client.0.smithi178.stdout: 2024-09-16T05:59:33.778 INFO:tasks.workunit.client.0.smithi178.stdout: services: 2024-09-16T05:59:33.778 INFO:tasks.workunit.client.0.smithi178.stdout: mon: 1 daemons, quorum a (age 100s) 2024-09-16T05:59:33.778 INFO:tasks.workunit.client.0.smithi178.stdout: mgr: no daemons active 2024-09-16T05:59:33.778 INFO:tasks.workunit.client.0.smithi178.stdout: osd: 3 osds: 3 up (since 5s), 3 in (since 35s) 2024-09-16T05:59:33.778 INFO:tasks.workunit.client.0.smithi178.stdout: 2024-09-16T05:59:33.778 INFO:tasks.workunit.client.0.smithi178.stdout: data: 2024-09-16T05:59:33.778 INFO:tasks.workunit.client.0.smithi178.stdout: pools: 0 pools, 0 pgs 2024-09-16T05:59:33.778 INFO:tasks.workunit.client.0.smithi178.stdout: objects: 0 objects, 0 B 2024-09-16T05:59:33.778 INFO:tasks.workunit.client.0.smithi178.stdout: usage: 0 B used, 0 B / 0 B avail 2024-09-16T05:59:33.778 INFO:tasks.workunit.client.0.smithi178.stdout: pgs: 2024-09-16T05:59:33.778 INFO:tasks.workunit.client.0.smithi178.stdout: 2024-09-16T05:59:33.786 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:516: test_run_mon: kill_daemons td/ceph-helpers 2024-09-16T05:59:33.787 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:59:33.787 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:59:33.787 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:59:33.787 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:59:33.787 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:59:39.100 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:59:39.100 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:518: test_run_mon: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:59:39.100 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T05:59:39.100 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:59:39.100 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:59:39.100 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:59:39.100 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T05:59:39.100 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:59:39.125 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:59:39.125+0000 7f0a56262c80 -1 'td/ceph-helpers/a' already exists and is not empty: monitor may already exist 2024-09-16T05:59:39.128 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:59:39.128 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:59:39.128 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:59:39.128 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:59:39.128 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:59:39.128 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:59:39.128 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:59:39.128 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T05:59:39.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:59:39.353 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:59:39.353 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:59:39.353 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:59:39.353 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:59:39.354 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:59:39.354 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:59:39.354 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:59:39.354 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:59:39.354 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:59:39.354 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:59:39.354 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:59:39.355 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:59:39.355 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:59:39.355 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T05:59:39.441 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:59:39.442 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:59:39.442 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:59:39.442 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:59:39.442 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:59:39.442 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:59:39.442 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:59:39.442 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:59:39.442 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:59:39.442 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:59:39.442 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:59:39.442 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:59:39.442 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:59:39.443 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T05:59:39.521 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: get_asok_path mon.a 2024-09-16T05:59:39.522 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:59:39.522 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:59:39.522 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:59:39.522 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:59:39.522 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:59:39.522 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:59:39.522 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: CEPH_ARGS= 2024-09-16T05:59:39.522 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: ceph --format=json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get osd_pool_default_size 2024-09-16T05:59:39.598 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: local 'size={"osd_pool_default_size":"1"}' 2024-09-16T05:59:39.599 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:521: test_run_mon: test '{"osd_pool_default_size":"1"}' = '{"osd_pool_default_size":"1"}' 2024-09-16T05:59:39.599 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:522: test_run_mon: kill_daemons td/ceph-helpers 2024-09-16T05:59:39.599 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:59:39.599 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:59:39.599 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:59:39.599 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:59:39.599 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:59:39.705 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:59:39.705 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:524: test_run_mon: CEPH_ARGS='--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd_pool_default_size=2' 2024-09-16T05:59:39.705 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:524: test_run_mon: run_mon td/ceph-helpers a 2024-09-16T05:59:39.705 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T05:59:39.705 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:59:39.705 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:59:39.705 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:59:39.705 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T05:59:39.705 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-16T05:59:39.727 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:59:39.726+0000 7f5cbe411c80 -1 'td/ceph-helpers/a' already exists and is not empty: monitor may already exist 2024-09-16T05:59:39.729 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:59:39.729 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:59:39.729 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:59:39.729 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:59:39.729 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:59:39.729 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:59:39.729 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:59:39.730 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T05:59:39.898 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:59:39.900 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:59:39.900 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:59:39.900 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:59:39.900 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:59:39.900 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:59:39.900 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:59:39.900 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:59:39.900 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:59:39.900 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:59:39.900 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:59:39.901 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:59:39.901 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:59:39.901 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:59:39.901 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T05:59:39.978 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:59:39.978 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:59:39.978 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:59:39.978 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:59:39.978 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:59:39.979 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:59:39.979 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:59:39.979 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:59:39.979 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:59:39.979 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:59:39.979 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:59:39.979 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:59:39.979 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:59:39.979 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T05:59:40.057 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: get_asok_path mon.a 2024-09-16T05:59:40.058 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:59:40.058 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:59:40.058 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:59:40.058 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:59:40.058 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:59:40.058 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:59:40.058 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: CEPH_ARGS= 2024-09-16T05:59:40.058 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: ceph --format=json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get osd_pool_default_size 2024-09-16T05:59:40.134 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: local 'size={"osd_pool_default_size":"2"}' 2024-09-16T05:59:40.135 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: test_run_mon: test '{"osd_pool_default_size":"2"}' = '{"osd_pool_default_size":"2"}' 2024-09-16T05:59:40.135 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:529: test_run_mon: kill_daemons td/ceph-helpers 2024-09-16T05:59:40.135 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:59:40.135 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:59:40.135 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:59:40.135 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:59:40.135 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:59:40.241 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:59:40.241 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:531: test_run_mon: teardown td/ceph-helpers 2024-09-16T05:59:40.241 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:59:40.241 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:59:40.241 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:59:40.242 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:59:40.242 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:59:40.242 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:59:40.242 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:59:40.242 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:59:40.245 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:59:40.245 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:59:40.246 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:59:40.246 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:59:40.247 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:59:40.247 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:59:40.247 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:59:40.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:59:40.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:59:40.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:59:40.249 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:59:40.249 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:59:40.250 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:59:40.250 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:59:40.261 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:59:40.261 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:59:40.261 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:59:40.261 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:59:40.263 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:59:40.263 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:59:40.263 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T05:59:40.263 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_run_osd td/ceph-helpers 2024-09-16T05:59:40.263 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:745: test_run_osd: local dir=td/ceph-helpers 2024-09-16T05:59:40.263 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:747: test_run_osd: setup td/ceph-helpers 2024-09-16T05:59:40.263 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T05:59:40.264 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T05:59:40.264 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T05:59:40.264 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T05:59:40.264 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T05:59:40.264 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T05:59:40.264 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T05:59:40.264 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T05:59:40.264 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T05:59:40.264 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T05:59:40.265 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T05:59:40.265 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T05:59:40.265 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T05:59:40.265 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T05:59:40.266 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T05:59:40.266 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T05:59:40.267 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T05:59:40.267 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:59:40.267 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T05:59:40.268 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T05:59:40.268 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T05:59:40.268 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T05:59:40.269 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T05:59:40.270 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T05:59:40.271 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T05:59:40.271 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:59:40.271 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:59:40.271 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T05:59:40.271 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T05:59:40.272 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T05:59:40.272 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T05:59:40.273 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T05:59:40.273 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:59:40.273 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:59:40.273 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T05:59:40.275 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T05:59:40.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T05:59:40.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T05:59:40.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T05:59:40.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:749: test_run_osd: run_mon td/ceph-helpers a 2024-09-16T05:59:40.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T05:59:40.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T05:59:40.276 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T05:59:40.276 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T05:59:40.276 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T05:59:40.276 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-16T05:59:40.651 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T05:59:40.651 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:59:40.651 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:59:40.652 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:59:40.652 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:59:40.652 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:59:40.652 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:59:40.652 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T05:59:40.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T05:59:40.844 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T05:59:40.844 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:59:40.844 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:59:40.844 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T05:59:40.844 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T05:59:40.845 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:59:40.845 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:59:40.845 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:59:40.845 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:59:40.846 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:59:40.846 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:59:40.846 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:59:40.846 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:59:40.846 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T05:59:40.925 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T05:59:40.925 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T05:59:40.925 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T05:59:40.926 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T05:59:40.926 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T05:59:40.926 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T05:59:40.926 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T05:59:40.926 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T05:59:40.926 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T05:59:40.926 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:59:40.926 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:59:40.926 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T05:59:40.926 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T05:59:40.926 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T05:59:41.003 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:750: test_run_osd: run_mgr td/ceph-helpers x 2024-09-16T05:59:41.004 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T05:59:41.004 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T05:59:41.004 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T05:59:41.004 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T05:59:41.004 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T05:59:41.004 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T05:59:41.849 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T05:59:41.849 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:59:41.849 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:59:41.849 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:59:41.849 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:59:41.849 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:59:41.850 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:59:41.850 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:59:41.850 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T05:59:41.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:752: test_run_osd: run_osd td/ceph-helpers 0 2024-09-16T05:59:41.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T05:59:41.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T05:59:41.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T05:59:41.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T05:59:41.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T05:59:41.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T05:59:41.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T05:59:41.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T05:59:41.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T05:59:41.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T05:59:41.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T05:59:41.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T05:59:41.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T05:59:41.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T05:59:41.882 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T05:59:41.882 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T05:59:41.883 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T05:59:41.883 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T05:59:41.883 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T05:59:41.883 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T05:59:41.883 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:59:41.883 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T05:59:41.883 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T05:59:41.883 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T05:59:41.883 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T05:59:41.883 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T05:59:41.883 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T05:59:41.883 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T05:59:41.883 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T05:59:41.884 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T05:59:41.884 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T05:59:41.884 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T05:59:41.884 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T05:59:41.884 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T05:59:41.884 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T05:59:41.884 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a95c968a-463e-477f-b305-65464f3e210a 2024-09-16T05:59:41.885 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 a95c968a-463e-477f-b305-65464f3e210a 2024-09-16T05:59:41.885 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a95c968a-463e-477f-b305-65464f3e210a' 2024-09-16T05:59:41.885 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T05:59:41.901 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBNyedmY4OpNRAAazJ2v4zhlcAkWUiIXeVlBQ== 2024-09-16T05:59:41.901 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBNyedmY4OpNRAAazJ2v4zhlcAkWUiIXeVlBQ=="}' 2024-09-16T05:59:41.901 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a95c968a-463e-477f-b305-65464f3e210a -i td/ceph-helpers/0/new.json 2024-09-16T05:59:42.167 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:59:42.177 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T05:59:42.178 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBNyedmY4OpNRAAazJ2v4zhlcAkWUiIXeVlBQ== --osd-uuid a95c968a-463e-477f-b305-65464f3e210a 2024-09-16T05:59:42.206 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:59:42.206+0000 7f976d945640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:59:42.236 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:59:42.236+0000 7f976d945640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:59:42.237 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:59:42.237+0000 7f976d945640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:59:42.238 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:59:42.237+0000 7f976d945640 -1 bdev(0x55b6803a7c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-16T05:59:42.238 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:59:42.238+0000 7f976d945640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T05:59:47.702 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T05:59:47.702 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T05:59:47.704 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T05:59:47.704 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T05:59:47.704 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T05:59:48.227 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T05:59:48.228 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T05:59:48.228 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T05:59:48.228 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T05:59:48.228 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T05:59:48.228 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T05:59:48.255 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:59:48.255+0000 7f68cf287640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:59:48.270 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:59:48.270+0000 7f68cf287640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:59:48.271 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:59:48.271+0000 7f68cf287640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T05:59:48.635 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T05:59:48.635 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T05:59:48.635 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T05:59:48.635 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T05:59:48.635 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T05:59:48.635 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:59:48.635 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T05:59:48.636 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T05:59:48.636 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:59:48.636 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:59:49.007 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:59:49.885 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:59:49.885+0000 7f68cf287640 -1 Falling back to public interface 2024-09-16T05:59:50.009 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:59:50.009 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:59:50.010 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T05:59:50.010 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T05:59:50.010 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:59:50.010 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:59:50.388 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:59:51.390 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:59:51.390 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:59:51.390 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T05:59:51.391 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T05:59:51.391 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:59:51.392 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:59:51.767 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:59:52.769 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:59:52.769 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:59:52.769 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T05:59:52.770 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T05:59:52.770 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:59:52.770 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:59:52.781 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T05:59:52.781+0000 7f68cf287640 -1 osd.0 0 log_to_monitors true 2024-09-16T05:59:53.225 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:59:54.226 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:59:54.226 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:59:54.226 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T05:59:54.227 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T05:59:54.227 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:59:54.227 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:59:54.686 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:59:55.687 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:59:55.688 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:59:55.688 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T05:59:55.688 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T05:59:55.688 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:59:55.689 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:59:56.065 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:59:57.067 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:59:57.067 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:59:57.067 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T05:59:57.068 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T05:59:57.068 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:59:57.068 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:59:57.509 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:59:58.510 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:59:58.510 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:59:58.511 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T05:59:58.511 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T05:59:58.511 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:59:58.511 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T05:59:58.913 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T05:59:59.914 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T05:59:59.915 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T05:59:59.915 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T05:59:59.915 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T05:59:59.916 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T05:59:59.916 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:00:00.351 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:00:01.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:00:01.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:00:01.354 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T06:00:01.355 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T06:00:01.355 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:00:01.355 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:00:01.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:00:02.884 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:00:02.884 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:00:02.885 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T06:00:02.885 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T06:00:02.885 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:00:02.885 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:00:03.287 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:00:04.289 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:00:04.289 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:00:04.289 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T06:00:04.289 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T06:00:04.290 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:00:04.290 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:00:04.672 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:00:05.673 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:00:05.673 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:00:05.674 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T06:00:05.674 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T06:00:05.674 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:00:05.674 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:00:06.717 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:00:07.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:00:07.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:00:07.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T06:00:07.719 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T06:00:07.720 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:00:07.720 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:00:08.721 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:00:09.722 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:00:09.722 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:00:09.722 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T06:00:09.723 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T06:00:09.723 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:00:09.723 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:00:10.649 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:00:11.650 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:00:11.651 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:00:11.651 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T06:00:11.651 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T06:00:11.651 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:00:11.651 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:00:12.225 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:00:13.227 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:00:13.227 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:00:13.227 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T06:00:13.228 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T06:00:13.228 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:00:13.228 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:00:13.974 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:00:14.975 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:00:14.975 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:00:14.975 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T06:00:14.976 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T06:00:14.976 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:00:14.976 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:00:15.760 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:00:16.761 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:00:16.762 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T06:00:16.762 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:00:16.762 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T06:00:16.762 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:00:16.763 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:00:17.215 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1990456247,v1:127.0.0.1:6803/1990456247] [v2:127.0.0.1:6804/1990456247,v1:127.0.0.1:6805/1990456247] exists,up a95c968a-463e-477f-b305-65464f3e210a 2024-09-16T06:00:17.216 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:00:17.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:00:17.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:00:17.217 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: get_asok_path osd.0 2024-09-16T06:00:17.217 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T06:00:17.217 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T06:00:17.217 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:00:17.217 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:00:17.217 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:00:17.217 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-osd.0.asok 2024-09-16T06:00:17.217 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: CEPH_ARGS= 2024-09-16T06:00:17.217 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: ceph --format=json daemon /tmp/ceph-asok.77187/ceph-osd.0.asok config get osd_max_backfills 2024-09-16T06:00:17.301 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: local 'backfills={"osd_max_backfills":"1"}' 2024-09-16T06:00:17.301 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:755: test_run_osd: echo '{"osd_max_backfills":"1"}' 2024-09-16T06:00:17.301 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:755: test_run_osd: grep --quiet osd_max_backfills 2024-09-16T06:00:17.305 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:757: test_run_osd: run_osd td/ceph-helpers 1 --osd-max-backfills 20 2024-09-16T06:00:17.305 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T06:00:17.305 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T06:00:17.305 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T06:00:17.305 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T06:00:17.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-09-16T06:00:17.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T06:00:17.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:00:17.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:00:17.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:00:17.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-16T06:00:17.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-16T06:00:17.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T06:00:17.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T06:00:17.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T06:00:17.306 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T06:00:17.306 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:00:17.306 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:00:17.306 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:00:17.307 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:00:17.307 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:00:17.307 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:00:17.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:00:17.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:00:17.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T06:00:17.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T06:00:17.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T06:00:17.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T06:00:17.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:00:17.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:00:17.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:00:17.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:00:17.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T06:00:17.308 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-max-backfills 20' 2024-09-16T06:00:17.308 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-09-16T06:00:17.308 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T06:00:17.308 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e02d6fd4-52eb-42c9-93f5-360b1b8906f8 2024-09-16T06:00:17.308 INFO:tasks.workunit.client.0.smithi178.stdout:add osd1 e02d6fd4-52eb-42c9-93f5-360b1b8906f8 2024-09-16T06:00:17.309 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 e02d6fd4-52eb-42c9-93f5-360b1b8906f8' 2024-09-16T06:00:17.309 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T06:00:17.324 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBxyedm+dBQExAAo5KNEKQDLkNhq5vTZ6zZaw== 2024-09-16T06:00:17.325 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBxyedm+dBQExAAo5KNEKQDLkNhq5vTZ6zZaw=="}' 2024-09-16T06:00:17.325 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e02d6fd4-52eb-42c9-93f5-360b1b8906f8 -i td/ceph-helpers/1/new.json 2024-09-16T06:00:17.770 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:00:17.781 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-09-16T06:00:17.782 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 --mkfs --key AQBxyedm+dBQExAAo5KNEKQDLkNhq5vTZ6zZaw== --osd-uuid e02d6fd4-52eb-42c9-93f5-360b1b8906f8 2024-09-16T06:00:17.811 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:00:17.811+0000 7fea8a20d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:00:17.832 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:00:17.831+0000 7fea8a20d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:00:17.833 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:00:17.832+0000 7fea8a20d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:00:17.833 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:00:17.833+0000 7fea8a20d640 -1 bdev(0x5557ae5cbc00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2024-09-16T06:00:17.833 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:00:17.833+0000 7fea8a20d640 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-16T06:00:23.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-16T06:00:23.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T06:00:23.785 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd1 key to auth repository 2024-09-16T06:00:23.785 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T06:00:23.785 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T06:00:24.302 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.1 2024-09-16T06:00:24.303 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T06:00:24.303 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2024-09-16T06:00:24.303 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T06:00:24.303 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T06:00:24.303 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T06:00:24.330 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:00:24.329+0000 7f9e52010640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:00:24.354 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:00:24.354+0000 7f9e52010640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:00:24.355 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:00:24.355+0000 7f9e52010640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:00:24.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T06:00:24.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:00:24.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T06:00:24.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:00:24.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:00:24.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:00:24.720 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:00:24.720 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:00:24.720 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:00:24.720 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:00:25.089 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:00:25.708 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:00:25.708+0000 7f9e52010640 -1 Falling back to public interface 2024-09-16T06:00:26.090 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:00:26.090 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:00:26.090 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:00:26.091 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T06:00:26.091 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:00:26.091 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:00:26.512 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:00:27.513 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:00:27.514 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:00:27.514 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:00:27.514 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T06:00:27.514 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:00:27.515 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:00:27.925 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:00:28.526 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:00:28.525+0000 7f9e52010640 -1 osd.1 0 log_to_monitors true 2024-09-16T06:00:28.926 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:00:28.926 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:00:28.927 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T06:00:28.927 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T06:00:28.927 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:00:28.927 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:00:29.366 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:00:30.367 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:00:30.367 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:00:30.368 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T06:00:30.368 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T06:00:30.368 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:00:30.368 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:00:30.774 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:00:31.775 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:00:31.775 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:00:31.776 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T06:00:31.776 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T06:00:31.776 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:00:31.776 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:00:32.274 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:00:33.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:00:33.276 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:00:33.276 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T06:00:33.276 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T06:00:33.277 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:00:33.277 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:00:33.665 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:00:34.667 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:00:34.667 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:00:34.667 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T06:00:34.667 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T06:00:34.668 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:00:34.668 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:00:35.075 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:00:36.077 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:00:36.077 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:00:36.077 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T06:00:36.078 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T06:00:36.078 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:00:36.079 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:00:36.615 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:00:37.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:00:37.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:00:37.617 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T06:00:37.618 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T06:00:37.619 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:00:37.619 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:00:38.038 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:00:39.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:00:39.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:00:39.040 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T06:00:39.040 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T06:00:39.040 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:00:39.040 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:00:39.524 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:00:40.525 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:00:40.525 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:00:40.526 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T06:00:40.526 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T06:00:40.526 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:00:40.526 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:00:41.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:00:42.027 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:00:42.027 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:00:42.028 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T06:00:42.028 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T06:00:42.028 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:00:42.028 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:00:42.766 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:00:43.767 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:00:43.768 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:00:43.768 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T06:00:43.768 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T06:00:43.768 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:00:43.768 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:00:44.604 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:00:45.605 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:00:45.606 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:00:45.606 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T06:00:45.606 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T06:00:45.607 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:00:45.607 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:00:46.368 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:00:47.369 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:00:47.370 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:00:47.370 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T06:00:47.371 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T06:00:47.371 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:00:47.371 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:00:48.082 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:00:49.083 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:00:49.083 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:00:49.083 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T06:00:49.084 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T06:00:49.084 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:00:49.084 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:00:49.907 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:00:50.909 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:00:50.909 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:00:50.909 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T06:00:50.910 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T06:00:50.910 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:00:50.910 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:00:52.083 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:00:53.085 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:00:53.085 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:00:53.085 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T06:00:53.085 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T06:00:53.086 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:00:53.086 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:00:53.508 INFO:tasks.workunit.client.0.smithi178.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4066595126,v1:127.0.0.1:6811/4066595126] [v2:127.0.0.1:6812/4066595126,v1:127.0.0.1:6813/4066595126] exists,up e02d6fd4-52eb-42c9-93f5-360b1b8906f8 2024-09-16T06:00:53.509 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:00:53.509 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:00:53.510 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:00:53.510 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: get_asok_path osd.1 2024-09-16T06:00:53.510 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T06:00:53.510 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T06:00:53.510 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:00:53.510 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:00:53.510 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:00:53.510 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-osd.1.asok 2024-09-16T06:00:53.510 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: CEPH_ARGS= 2024-09-16T06:00:53.510 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: ceph --format=json daemon /tmp/ceph-asok.77187/ceph-osd.1.asok config get osd_max_backfills 2024-09-16T06:00:53.597 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: local 'backfills={"osd_max_backfills":"20"}' 2024-09-16T06:00:53.597 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:760: test_run_osd: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2024-09-16T06:00:53.597 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:762: test_run_osd: CEPH_ARGS='--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30' 2024-09-16T06:00:53.597 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:762: test_run_osd: run_osd td/ceph-helpers 2 2024-09-16T06:00:53.597 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T06:00:53.597 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T06:00:53.597 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T06:00:53.598 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T06:00:53.598 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/2 2024-09-16T06:00:53.598 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30' 2024-09-16T06:00:53.598 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:00:53.598 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:00:53.598 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:00:53.598 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2024-09-16T06:00:53.598 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2024-09-16T06:00:53.599 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T06:00:53.599 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T06:00:53.599 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T06:00:53.599 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T06:00:53.599 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:00:53.599 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:00:53.599 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:00:53.599 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:00:53.599 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:00:53.599 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:00:53.599 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:00:53.599 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:00:53.600 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T06:00:53.600 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T06:00:53.600 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T06:00:53.600 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T06:00:53.600 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:00:53.600 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:00:53.600 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:00:53.600 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:00:53.600 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T06:00:53.600 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T06:00:53.601 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/2 2024-09-16T06:00:53.601 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T06:00:53.601 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ba702f2d-7291-4a97-b3a7-7d4e636a840c 2024-09-16T06:00:53.601 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 ba702f2d-7291-4a97-b3a7-7d4e636a840c' 2024-09-16T06:00:53.601 INFO:tasks.workunit.client.0.smithi178.stdout:add osd2 ba702f2d-7291-4a97-b3a7-7d4e636a840c 2024-09-16T06:00:53.601 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T06:00:53.616 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCVyedmJxO4JBAA6lgL61zgL4Jj92MeWeTS+Q== 2024-09-16T06:00:53.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCVyedmJxO4JBAA6lgL61zgL4Jj92MeWeTS+Q=="}' 2024-09-16T06:00:53.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ba702f2d-7291-4a97-b3a7-7d4e636a840c -i td/ceph-helpers/2/new.json 2024-09-16T06:00:54.061 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:00:54.071 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/2/new.json 2024-09-16T06:00:54.072 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCVyedmJxO4JBAA6lgL61zgL4Jj92MeWeTS+Q== --osd-uuid ba702f2d-7291-4a97-b3a7-7d4e636a840c 2024-09-16T06:00:54.103 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:00:54.103+0000 7f6271815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:00:54.132 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:00:54.131+0000 7f6271815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:00:54.132 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:00:54.132+0000 7f6271815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:00:54.133 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:00:54.133+0000 7f6271815640 -1 bdev(0x55b1113d6800 td/ceph-helpers/2/block) open stat got: (1) Operation not permitted 2024-09-16T06:00:54.133 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:00:54.133+0000 7f6271815640 -1 bluestore(td/ceph-helpers/2) _read_fsid unparsable uuid 2024-09-16T06:01:00.049 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/2/keyring 2024-09-16T06:01:00.049 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T06:01:00.050 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd2 key to auth repository 2024-09-16T06:01:00.050 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T06:01:00.050 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T06:01:00.564 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T06:01:00.565 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.2 2024-09-16T06:01:00.565 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T06:01:00.565 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T06:01:00.566 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T06:01:00.566 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T06:01:00.593 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:01:00.593+0000 7ffbf0f34640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:01:00.595 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:01:00.595+0000 7ffbf0f34640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:01:00.596 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:01:00.596+0000 7ffbf0f34640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:01:00.937 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T06:01:00.938 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:01:00.938 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T06:01:00.938 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:01:00.938 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:01:00.938 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:01:00.938 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:01:00.939 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:01:00.939 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:01:00.940 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:01:01.311 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:01:01.702 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:01:01.702+0000 7ffbf0f34640 -1 Falling back to public interface 2024-09-16T06:01:02.312 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:01:02.312 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:01:02.312 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:01:02.313 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T06:01:02.313 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:01:02.313 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:01:02.722 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:01:03.724 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:01:03.724 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:01:03.724 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:01:03.725 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T06:01:03.725 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:01:03.725 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:01:04.145 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:01:04.854 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:01:04.853+0000 7ffbf0f34640 -1 osd.2 0 log_to_monitors true 2024-09-16T06:01:05.146 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:01:05.146 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:01:05.146 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T06:01:05.147 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T06:01:05.147 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:01:05.147 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:01:05.664 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:01:06.665 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:01:06.665 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:01:06.666 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T06:01:06.666 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T06:01:06.666 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:01:06.666 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:01:07.061 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:01:08.063 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:01:08.063 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T06:01:08.064 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:01:08.064 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T06:01:08.064 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:01:08.064 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:01:08.477 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:01:09.479 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:01:09.479 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:01:09.479 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T06:01:09.480 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T06:01:09.480 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:01:09.480 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:01:10.017 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:01:11.018 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:01:11.018 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:01:11.018 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T06:01:11.019 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T06:01:11.019 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:01:11.019 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:01:11.444 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:01:12.445 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:01:12.445 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:01:12.445 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T06:01:12.446 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T06:01:12.446 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:01:12.446 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:01:12.929 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:01:13.930 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:01:13.931 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:01:13.931 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T06:01:13.931 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T06:01:13.932 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:01:13.932 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:01:14.345 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:01:15.346 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:01:15.346 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:01:15.347 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T06:01:15.347 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T06:01:15.347 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:01:15.347 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:01:15.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:01:16.820 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:01:16.821 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:01:16.821 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T06:01:16.821 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T06:01:16.822 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:01:16.822 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:01:17.265 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:01:18.267 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:01:18.267 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:01:18.267 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T06:01:18.268 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T06:01:18.268 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:01:18.268 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:01:19.496 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:01:20.498 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:01:20.498 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:01:20.498 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T06:01:20.499 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T06:01:20.499 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:01:20.499 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:01:20.965 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:01:21.966 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:01:21.967 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T06:01:21.967 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:01:21.967 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T06:01:21.967 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:01:21.967 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:01:22.901 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:01:23.902 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:01:23.902 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:01:23.903 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T06:01:23.903 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T06:01:23.903 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:01:23.903 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:01:24.826 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:01:25.828 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:01:25.828 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:01:25.828 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T06:01:25.828 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T06:01:25.829 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:01:25.829 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:01:26.603 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:01:27.604 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:01:27.604 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:01:27.604 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T06:01:27.605 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T06:01:27.605 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:01:27.605 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:01:28.278 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:01:29.279 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:01:29.279 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:01:29.280 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T06:01:29.280 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T06:01:29.280 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:01:29.280 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:01:29.659 INFO:tasks.workunit.client.0.smithi178.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4205007912,v1:127.0.0.1:6819/4205007912] [v2:127.0.0.1:6820/4205007912,v1:127.0.0.1:6821/4205007912] exists,up ba702f2d-7291-4a97-b3a7-7d4e636a840c 2024-09-16T06:01:29.660 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:01:29.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:01:29.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:01:29.661 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: get_asok_path osd.2 2024-09-16T06:01:29.661 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-16T06:01:29.661 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-16T06:01:29.661 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:01:29.661 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:01:29.661 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:01:29.661 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-osd.2.asok 2024-09-16T06:01:29.661 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: CEPH_ARGS= 2024-09-16T06:01:29.661 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: ceph --format=json daemon /tmp/ceph-asok.77187/ceph-osd.2.asok config get osd_max_backfills 2024-09-16T06:01:29.746 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: local 'backfills={"osd_max_backfills":"30"}' 2024-09-16T06:01:29.746 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:765: test_run_osd: test '{"osd_max_backfills":"30"}' = '{"osd_max_backfills":"30"}' 2024-09-16T06:01:29.746 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:767: test_run_osd: teardown td/ceph-helpers 2024-09-16T06:01:29.746 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T06:01:29.746 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T06:01:29.746 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T06:01:29.746 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:01:29.746 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:01:29.746 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:01:29.747 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:01:29.747 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:01:29.853 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:01:29.853 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:01:29.854 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:01:29.854 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:01:29.855 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:01:29.855 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:01:29.855 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:01:29.856 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:01:29.856 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:01:29.856 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:01:29.857 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:01:29.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:01:29.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T06:01:29.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T06:01:29.870 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:01:29.870 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:01:29.870 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:01:29.871 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T06:01:29.872 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:01:29.872 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:01:29.873 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T06:01:29.873 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_set_config td/ceph-helpers 2024-09-16T06:01:29.873 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: test_set_config: local dir=td/ceph-helpers 2024-09-16T06:01:29.873 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: test_set_config: setup td/ceph-helpers 2024-09-16T06:01:29.873 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T06:01:29.873 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T06:01:29.873 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T06:01:29.873 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T06:01:29.873 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T06:01:29.873 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:01:29.873 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:01:29.873 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:01:29.873 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:01:29.874 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:01:29.874 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:01:29.874 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:01:29.875 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:01:29.875 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:01:29.876 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:01:29.876 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:01:29.876 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:01:29.877 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:01:29.877 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:01:29.877 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:01:29.877 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:01:29.878 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:01:29.879 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T06:01:29.879 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T06:01:29.880 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:01:29.880 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:01:29.880 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:01:29.880 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T06:01:29.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:01:29.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:01:29.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T06:01:29.882 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T06:01:29.882 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:01:29.882 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:01:29.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T06:01:29.883 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T06:01:29.883 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T06:01:29.885 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T06:01:29.885 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T06:01:29.885 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: test_set_config: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T06:01:29.885 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T06:01:29.885 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T06:01:29.885 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T06:01:29.885 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T06:01:29.885 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T06:01:29.885 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T06:01:30.243 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T06:01:30.243 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:01:30.243 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:01:30.243 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:01:30.243 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:01:30.244 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:01:30.244 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:01:30.244 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T06:01:30.452 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T06:01:30.454 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T06:01:30.454 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T06:01:30.455 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T06:01:30.455 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T06:01:30.455 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T06:01:30.455 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T06:01:30.455 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T06:01:30.455 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T06:01:30.455 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:01:30.455 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:01:30.455 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:01:30.456 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T06:01:30.456 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T06:01:30.456 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T06:01:30.534 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T06:01:30.534 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T06:01:30.535 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T06:01:30.535 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T06:01:30.535 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T06:01:30.535 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T06:01:30.535 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T06:01:30.535 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T06:01:30.536 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:01:30.536 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:01:30.536 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:01:30.536 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T06:01:30.536 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T06:01:30.536 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T06:01:30.612 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: test_set_config: get_config mon a ms_crc_header 2024-09-16T06:01:30.612 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T06:01:30.612 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T06:01:30.612 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=ms_crc_header 2024-09-16T06:01:30.613 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .ms_crc_header 2024-09-16T06:01:30.614 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T06:01:30.614 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T06:01:30.614 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T06:01:30.614 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:01:30.614 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:01:30.614 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:01:30.614 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T06:01:30.614 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T06:01:30.614 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get ms_crc_header 2024-09-16T06:01:30.692 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: test_set_config: test true = true 2024-09-16T06:01:30.692 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1177: test_set_config: set_config mon a ms_crc_header false 2024-09-16T06:01:30.692 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=mon 2024-09-16T06:01:30.692 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=a 2024-09-16T06:01:30.692 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=ms_crc_header 2024-09-16T06:01:30.692 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=false 2024-09-16T06:01:30.692 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T06:01:30.693 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path mon.a 2024-09-16T06:01:30.693 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T06:01:30.693 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T06:01:30.693 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:01:30.693 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:01:30.693 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:01:30.693 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T06:01:30.693 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config set ms_crc_header false 2024-09-16T06:01:30.771 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T06:01:30.771 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1178: test_set_config: get_config mon a ms_crc_header 2024-09-16T06:01:30.771 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T06:01:30.771 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T06:01:30.771 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=ms_crc_header 2024-09-16T06:01:30.771 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .ms_crc_header 2024-09-16T06:01:30.771 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T06:01:30.771 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T06:01:30.772 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T06:01:30.772 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:01:30.772 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:01:30.772 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:01:30.772 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T06:01:30.772 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T06:01:30.772 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get ms_crc_header 2024-09-16T06:01:30.850 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1178: test_set_config: test false = false 2024-09-16T06:01:30.850 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1179: test_set_config: set_config mon a ms_crc_header true 2024-09-16T06:01:30.850 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=mon 2024-09-16T06:01:30.850 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=a 2024-09-16T06:01:30.850 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=ms_crc_header 2024-09-16T06:01:30.850 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T06:01:30.850 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T06:01:30.850 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path mon.a 2024-09-16T06:01:30.850 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T06:01:30.850 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T06:01:30.851 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:01:30.851 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:01:30.851 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:01:30.851 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T06:01:30.851 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config set ms_crc_header true 2024-09-16T06:01:30.929 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T06:01:30.929 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1180: test_set_config: get_config mon a ms_crc_header 2024-09-16T06:01:30.929 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T06:01:30.929 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T06:01:30.930 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=ms_crc_header 2024-09-16T06:01:30.930 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .ms_crc_header 2024-09-16T06:01:30.930 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T06:01:30.930 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T06:01:30.930 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T06:01:30.930 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:01:30.930 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:01:30.930 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:01:30.930 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T06:01:30.931 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T06:01:30.931 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get ms_crc_header 2024-09-16T06:01:31.009 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1180: test_set_config: test true = true 2024-09-16T06:01:31.009 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1181: test_set_config: teardown td/ceph-helpers 2024-09-16T06:01:31.009 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T06:01:31.009 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T06:01:31.009 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T06:01:31.009 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:01:31.009 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:01:31.009 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:01:31.009 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:01:31.009 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:01:31.113 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:01:31.113 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:01:31.114 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:01:31.114 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:01:31.115 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:01:31.115 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:01:31.115 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:01:31.116 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:01:31.116 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:01:31.117 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:01:31.117 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:01:31.117 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:01:31.118 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T06:01:31.118 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T06:01:31.120 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:01:31.120 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:01:31.120 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:01:31.120 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T06:01:31.122 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:01:31.122 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:01:31.122 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T06:01:31.122 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_setup td/ceph-helpers 2024-09-16T06:01:31.122 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:144: test_setup: local dir=td/ceph-helpers 2024-09-16T06:01:31.122 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:145: test_setup: setup td/ceph-helpers 2024-09-16T06:01:31.122 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T06:01:31.122 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T06:01:31.122 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T06:01:31.123 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T06:01:31.123 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T06:01:31.123 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:01:31.123 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:01:31.123 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:01:31.123 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:01:31.123 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:01:31.124 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:01:31.124 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:01:31.124 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:01:31.124 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:01:31.125 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:01:31.125 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:01:31.126 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:01:31.126 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:01:31.126 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:01:31.127 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:01:31.127 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:01:31.127 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:01:31.128 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T06:01:31.129 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T06:01:31.130 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:01:31.130 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:01:31.130 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:01:31.130 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T06:01:31.131 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:01:31.131 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:01:31.131 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T06:01:31.132 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T06:01:31.132 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:01:31.132 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:01:31.132 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T06:01:31.134 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T06:01:31.135 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T06:01:31.135 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T06:01:31.135 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T06:01:31.135 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:146: test_setup: test -d td/ceph-helpers 2024-09-16T06:01:31.135 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:147: test_setup: setup td/ceph-helpers 2024-09-16T06:01:31.135 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T06:01:31.135 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T06:01:31.135 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T06:01:31.135 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T06:01:31.135 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T06:01:31.135 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:01:31.135 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:01:31.135 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:01:31.136 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:01:31.136 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:01:31.136 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:01:31.136 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:01:31.137 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:01:31.137 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:01:31.138 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:01:31.138 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:01:31.138 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:01:31.139 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:01:31.139 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:01:31.140 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:01:31.140 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:01:31.140 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:01:31.141 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T06:01:31.141 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T06:01:31.142 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:01:31.142 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:01:31.142 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:01:31.143 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T06:01:31.143 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:01:31.143 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:01:31.143 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T06:01:31.145 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T06:01:31.145 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:01:31.145 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:01:31.145 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T06:01:31.147 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T06:01:31.147 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T06:01:31.147 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T06:01:31.147 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T06:01:31.147 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:148: test_setup: test -d td/ceph-helpers 2024-09-16T06:01:31.147 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:149: test_setup: teardown td/ceph-helpers 2024-09-16T06:01:31.147 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T06:01:31.147 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T06:01:31.147 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T06:01:31.148 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:01:31.148 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:01:31.148 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:01:31.148 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:01:31.148 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:01:31.148 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:01:31.148 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:01:31.149 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:01:31.149 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:01:31.150 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:01:31.150 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:01:31.151 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:01:31.152 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:01:31.152 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:01:31.152 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:01:31.152 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:01:31.152 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:01:31.153 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T06:01:31.154 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T06:01:31.155 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:01:31.155 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:01:31.155 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:01:31.155 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T06:01:31.157 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:01:31.157 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:01:31.157 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T06:01:31.157 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_teardown td/ceph-helpers 2024-09-16T06:01:31.157 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:220: test_teardown: local dir=td/ceph-helpers 2024-09-16T06:01:31.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:221: test_teardown: setup td/ceph-helpers 2024-09-16T06:01:31.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T06:01:31.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T06:01:31.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T06:01:31.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T06:01:31.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T06:01:31.158 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:01:31.158 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:01:31.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:01:31.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:01:31.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:01:31.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:01:31.159 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:01:31.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:01:31.159 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:01:31.160 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:01:31.160 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:01:31.160 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:01:31.161 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:01:31.161 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:01:31.162 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:01:31.162 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:01:31.162 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:01:31.163 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T06:01:31.163 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T06:01:31.164 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:01:31.164 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:01:31.165 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:01:31.165 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T06:01:31.165 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:01:31.165 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:01:31.165 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T06:01:31.167 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T06:01:31.167 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:01:31.167 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:01:31.167 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T06:01:31.169 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T06:01:31.169 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T06:01:31.169 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T06:01:31.169 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T06:01:31.169 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:222: test_teardown: teardown td/ceph-helpers 2024-09-16T06:01:31.169 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T06:01:31.169 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T06:01:31.169 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T06:01:31.169 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:01:31.169 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:01:31.169 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:01:31.170 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:01:31.170 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:01:31.170 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:01:31.170 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:01:31.171 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:01:31.171 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:01:31.172 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:01:31.172 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:01:31.173 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:01:31.174 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:01:31.174 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:01:31.174 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:01:31.174 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:01:31.174 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:01:31.175 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T06:01:31.176 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T06:01:31.177 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:01:31.177 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:01:31.177 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:01:31.177 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T06:01:31.180 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:01:31.180 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:01:31.180 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:223: test_teardown: test -d td/ceph-helpers 2024-09-16T06:01:31.180 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T06:01:31.180 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_wait_background td/ceph-helpers 2024-09-16T06:01:31.180 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: test_wait_background: local pids= 2024-09-16T06:01:31.180 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: test_wait_background: run_in_background pids bash -c 'sleep 1; exit 1' 2024-09-16T06:01:31.180 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-09-16T06:01:31.180 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-09-16T06:01:31.180 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 150772"' 2024-09-16T06:01:31.180 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 150772' 2024-09-16T06:01:31.180 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: test_wait_background: run_in_background pids bash -c 'sleep 2; exit 0' 2024-09-16T06:01:31.181 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-09-16T06:01:31.181 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-09-16T06:01:31.181 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 150774"' 2024-09-16T06:01:31.181 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: bash -c 'sleep 1; exit 1' 2024-09-16T06:01:31.181 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 150774' 2024-09-16T06:01:31.181 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: test_wait_background: wait_background pids 2024-09-16T06:01:31.181 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 150772 150774' 2024-09-16T06:01:31.181 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-09-16T06:01:31.181 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/150775: /' 2024-09-16T06:01:31.181 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-09-16T06:01:31.181 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 150772 2024-09-16T06:01:31.181 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: bash -c 'sleep 2; exit 0' 2024-09-16T06:01:31.181 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/150777: /' 2024-09-16T06:01:32.181 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 1 2024-09-16T06:01:32.181 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: wait_background: return_code=1 2024-09-16T06:01:32.181 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-09-16T06:01:32.181 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 150774 2024-09-16T06:01:33.183 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-09-16T06:01:33.183 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-09-16T06:01:33.183 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-09-16T06:01:33.183 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 1 2024-09-16T06:01:33.183 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: test_wait_background: '[' 1 -ne 1 ']' 2024-09-16T06:01:33.183 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: test_wait_background: run_in_background pids bash -c 'sleep 1; exit 0' 2024-09-16T06:01:33.183 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-09-16T06:01:33.184 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-09-16T06:01:33.184 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 150780"' 2024-09-16T06:01:33.184 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 150780' 2024-09-16T06:01:33.184 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: test_wait_background: run_in_background pids bash -c 'sleep 2; exit 0' 2024-09-16T06:01:33.184 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-09-16T06:01:33.184 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-09-16T06:01:33.184 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: bash -c 'sleep 1; exit 0' 2024-09-16T06:01:33.184 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 150782"' 2024-09-16T06:01:33.184 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 150782' 2024-09-16T06:01:33.184 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2189: test_wait_background: wait_background pids 2024-09-16T06:01:33.184 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/150783: /' 2024-09-16T06:01:33.184 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 150780 150782' 2024-09-16T06:01:33.184 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-09-16T06:01:33.185 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-09-16T06:01:33.185 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 150780 2024-09-16T06:01:33.185 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: bash -c 'sleep 2; exit 0' 2024-09-16T06:01:33.185 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/150785: /' 2024-09-16T06:01:34.185 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-09-16T06:01:34.185 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-09-16T06:01:34.185 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 150782 2024-09-16T06:01:35.186 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-09-16T06:01:35.187 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-09-16T06:01:35.187 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-09-16T06:01:35.187 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-09-16T06:01:35.187 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: test_wait_background: '[' 0 -ne 0 ']' 2024-09-16T06:01:35.187 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2192: test_wait_background: '[' '!' -z '' ']' 2024-09-16T06:01:35.187 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T06:01:35.187 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_wait_for_clean td/ceph-helpers 2024-09-16T06:01:35.187 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1677: test_wait_for_clean: local dir=td/ceph-helpers 2024-09-16T06:01:35.187 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1679: test_wait_for_clean: setup td/ceph-helpers 2024-09-16T06:01:35.187 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T06:01:35.187 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T06:01:35.187 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T06:01:35.188 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T06:01:35.188 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T06:01:35.188 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:01:35.188 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:01:35.188 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:01:35.188 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:01:35.188 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:01:35.189 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:01:35.189 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:01:35.189 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:01:35.189 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:01:35.191 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:01:35.191 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:01:35.191 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:01:35.192 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:01:35.192 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:01:35.192 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:01:35.193 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:01:35.193 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:01:35.195 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T06:01:35.195 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T06:01:35.196 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:01:35.196 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:01:35.196 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:01:35.196 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T06:01:35.198 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:01:35.198 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:01:35.198 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T06:01:35.199 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T06:01:35.199 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:01:35.199 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:01:35.199 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T06:01:35.202 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T06:01:35.202 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T06:01:35.202 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T06:01:35.202 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T06:01:35.202 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1680: test_wait_for_clean: run_mon td/ceph-helpers a --osd_pool_default_size=2 2024-09-16T06:01:35.202 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T06:01:35.202 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T06:01:35.202 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T06:01:35.203 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T06:01:35.203 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T06:01:35.203 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2024-09-16T06:01:35.590 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T06:01:35.590 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:01:35.590 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:01:35.590 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:01:35.590 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:01:35.591 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:01:35.591 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:01:35.591 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-16T06:01:35.774 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T06:01:35.775 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T06:01:35.775 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T06:01:35.775 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T06:01:35.775 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T06:01:35.776 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T06:01:35.777 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T06:01:35.777 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T06:01:35.777 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T06:01:35.777 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:01:35.777 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:01:35.777 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:01:35.777 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T06:01:35.777 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T06:01:35.777 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T06:01:35.864 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T06:01:35.865 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T06:01:35.865 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T06:01:35.865 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T06:01:35.865 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T06:01:35.865 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T06:01:35.865 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T06:01:35.865 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T06:01:35.865 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:01:35.865 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:01:35.865 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:01:35.865 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T06:01:35.866 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T06:01:35.866 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T06:01:35.948 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1681: test_wait_for_clean: run_osd td/ceph-helpers 0 2024-09-16T06:01:35.948 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T06:01:35.949 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T06:01:35.949 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T06:01:35.949 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T06:01:35.949 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T06:01:35.949 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T06:01:35.949 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:01:35.950 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:01:35.950 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:01:35.950 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T06:01:35.950 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T06:01:35.950 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T06:01:35.950 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T06:01:35.950 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T06:01:35.951 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T06:01:35.951 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:01:35.951 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:01:35.951 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:01:35.951 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:01:35.951 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:01:35.951 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:01:35.952 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:01:35.952 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:01:35.952 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T06:01:35.952 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T06:01:35.952 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T06:01:35.952 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T06:01:35.953 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:01:35.953 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:01:35.953 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:01:35.953 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:01:35.953 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T06:01:35.953 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T06:01:35.954 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T06:01:35.954 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T06:01:35.954 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6189a740-f623-496b-81d5-8c76481ac225 2024-09-16T06:01:35.954 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 6189a740-f623-496b-81d5-8c76481ac225 2024-09-16T06:01:35.955 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6189a740-f623-496b-81d5-8c76481ac225' 2024-09-16T06:01:35.955 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T06:01:35.966 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC/yedmBj6UORAAw4ZeeIs4m5lQichAwrb4rw== 2024-09-16T06:01:35.966 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC/yedmBj6UORAAw4ZeeIs4m5lQichAwrb4rw=="}' 2024-09-16T06:01:35.966 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6189a740-f623-496b-81d5-8c76481ac225 -i td/ceph-helpers/0/new.json 2024-09-16T06:01:36.823 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:01:36.832 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T06:01:36.833 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC/yedmBj6UORAAw4ZeeIs4m5lQichAwrb4rw== --osd-uuid 6189a740-f623-496b-81d5-8c76481ac225 2024-09-16T06:01:36.861 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:01:36.860+0000 7f90a5c83640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:01:36.891 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:01:36.890+0000 7f90a5c83640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:01:36.892 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:01:36.891+0000 7f90a5c83640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:01:36.892 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:01:36.892+0000 7f90a5c83640 -1 bdev(0x55d90ecedc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-16T06:01:36.892 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:01:36.892+0000 7f90a5c83640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T06:01:42.372 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T06:01:42.372 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T06:01:42.373 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T06:01:42.374 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T06:01:42.374 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T06:01:42.577 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T06:01:42.578 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T06:01:42.578 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T06:01:42.578 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T06:01:42.578 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T06:01:42.578 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T06:01:42.604 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:01:42.603+0000 7f6aeaa2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:01:42.614 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:01:42.614+0000 7f6aeaa2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:01:42.615 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:01:42.615+0000 7f6aeaa2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:01:42.788 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T06:01:42.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:01:42.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T06:01:42.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:01:42.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:01:42.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:01:42.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:01:42.789 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:01:42.790 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:01:42.790 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:01:43.005 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:01:44.006 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:01:44.006 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:01:44.006 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:01:44.007 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T06:01:44.007 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:01:44.007 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:01:44.203 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:01:44.216 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:01:44.216+0000 7f6aeaa2e640 -1 Falling back to public interface 2024-09-16T06:01:45.205 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:01:45.205 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:01:45.205 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:01:45.205 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T06:01:45.206 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:01:45.206 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:01:45.399 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:01:46.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:01:46.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:01:46.401 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T06:01:46.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T06:01:46.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:01:46.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:01:46.593 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:01:46.948 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:01:46.946+0000 7f6aeaa2e640 -1 osd.0 0 log_to_monitors true 2024-09-16T06:01:47.594 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:01:47.594 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T06:01:47.595 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:01:47.595 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T06:01:47.595 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:01:47.595 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:01:47.985 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:01:48.986 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:01:48.987 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:01:48.987 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T06:01:48.987 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T06:01:48.987 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:01:48.988 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:01:49.179 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:01:50.180 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:01:50.180 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:01:50.181 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T06:01:50.181 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T06:01:50.181 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:01:50.181 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:01:50.376 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:01:51.378 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:01:51.378 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:01:51.378 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T06:01:51.379 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T06:01:51.379 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:01:51.379 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:01:51.576 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:01:52.577 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:01:52.577 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:01:52.577 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T06:01:52.578 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T06:01:52.578 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:01:52.578 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:01:52.773 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:01:53.774 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:01:53.774 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:01:53.775 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T06:01:53.775 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T06:01:53.775 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:01:53.775 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:01:53.971 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:01:54.972 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:01:54.973 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:01:54.973 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T06:01:54.973 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T06:01:54.973 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:01:54.973 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:01:55.167 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:01:56.168 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:01:56.168 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:01:56.169 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T06:01:56.169 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T06:01:56.169 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:01:56.169 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:01:56.367 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:01:57.368 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:01:57.368 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:01:57.369 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T06:01:57.369 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T06:01:57.369 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:01:57.369 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:01:57.564 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:01:58.565 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:01:58.565 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:01:58.566 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T06:01:58.566 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T06:01:58.566 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:01:58.566 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:01:58.764 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:01:59.765 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:01:59.766 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:01:59.766 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T06:01:59.766 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T06:01:59.766 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:01:59.767 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:01:59.953 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:02:00.954 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:02:00.954 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:02:00.954 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T06:02:00.955 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T06:02:00.955 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:02:00.955 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:02:01.152 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:02:02.153 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:02:02.153 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:02:02.154 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T06:02:02.154 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T06:02:02.154 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:02:02.154 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:02:02.349 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:02:03.351 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:02:03.351 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:02:03.351 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T06:02:03.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T06:02:03.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:02:03.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:02:03.540 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:02:04.541 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:02:04.542 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:02:04.542 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T06:02:04.542 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T06:02:04.542 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:02:04.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:02:04.738 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:02:05.740 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:02:05.740 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:02:05.740 INFO:tasks.workunit.client.0.smithi178.stdout:19 2024-09-16T06:02:05.741 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-16T06:02:05.741 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:02:05.741 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:02:05.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:02:06.934 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:02:06.934 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:02:06.935 INFO:tasks.workunit.client.0.smithi178.stdout:20 2024-09-16T06:02:06.935 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-16T06:02:06.935 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:02:06.935 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:02:07.129 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:02:08.130 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:02:08.130 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:02:08.131 INFO:tasks.workunit.client.0.smithi178.stdout:21 2024-09-16T06:02:08.131 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-09-16T06:02:08.131 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:02:08.131 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:02:08.351 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1284002052,v1:127.0.0.1:6801/1284002052] [v2:127.0.0.1:6802/1284002052,v1:127.0.0.1:6803/1284002052] exists,up 6189a740-f623-496b-81d5-8c76481ac225 2024-09-16T06:02:08.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:02:08.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:02:08.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:02:08.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1682: test_wait_for_clean: run_mgr td/ceph-helpers x 2024-09-16T06:02:08.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T06:02:08.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T06:02:08.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T06:02:08.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T06:02:08.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T06:02:08.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T06:02:08.598 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T06:02:08.599 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:02:08.599 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:02:08.599 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:02:08.599 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:02:08.599 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:02:08.599 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:02:08.600 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T06:02:08.600 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T06:02:08.627 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1683: test_wait_for_clean: create_rbd_pool 2024-09-16T06:02:08.627 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T06:02:08.932 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' does not exist 2024-09-16T06:02:08.942 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T06:02:08.942 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T06:02:09.303 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' already exists 2024-09-16T06:02:09.313 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T06:02:10.314 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T06:02:11.315 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: test_wait_for_clean: WAIT_FOR_CLEAN_TIMEOUT=1 2024-09-16T06:02:11.316 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: test_wait_for_clean: wait_for_clean 2024-09-16T06:02:11.316 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T06:02:11.316 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T06:02:11.316 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T06:02:11.316 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 1 .1 2024-09-16T06:02:11.316 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T06:02:11.316 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T06:02:11.316 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T06:02:11.316 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T06:02:11.316 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T06:02:11.343 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.3') 2024-09-16T06:02:11.343 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T06:02:11.343 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T06:02:11.343 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T06:02:11.343 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-16T06:02:11.343 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-16T06:02:11.550 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-09-16T06:02:11.550 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-16T06:02:11.550 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T06:02:11.550 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T06:02:11.684 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=0 2024-09-16T06:02:11.685 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 0 2024-09-16T06:02:11.685 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-0' 2024-09-16T06:02:11.685 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T06:02:11.685 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-0 2024-09-16T06:02:11.685 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T06:02:11.686 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-16T06:02:11.687 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-0 2024-09-16T06:02:11.687 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T06:02:11.688 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.0 seq 0 2024-09-16T06:02:11.688 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=0 2024-09-16T06:02:11.688 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 0' 2024-09-16T06:02:11.689 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T06:02:11.877 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 0 -lt 0 2024-09-16T06:02:11.878 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T06:02:11.878 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T06:02:11.878 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T06:02:12.082 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 == 0 2024-09-16T06:02:12.083 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: sleep 1 2024-09-16T06:02:13.084 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T06:02:13.084 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T06:02:13.085 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T06:02:13.589 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T06:02:13.589 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T06:02:13.589 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T06:02:13.589 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T06:02:13.589 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T06:02:13.589 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T06:02:13.589 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T06:02:13.589 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T06:02:13.953 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T06:02:13.953 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T06:02:13.953 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T06:02:13.953 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T06:02:14.452 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-16T06:02:14.453 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-16T06:02:14.453 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-16T06:02:14.453 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-16T06:02:14.453 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T06:02:14.453 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T06:02:14.551 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T06:02:14.551 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T06:02:14.552 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T06:02:14.552 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T06:02:14.552 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T06:02:14.552 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T06:02:14.552 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T06:02:14.552 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T06:02:14.928 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T06:02:14.928 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T06:02:14.929 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T06:02:14.929 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T06:02:15.438 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-16T06:02:15.438 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T06:02:15.438 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T06:02:15.438 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T06:02:15.438 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T06:02:15.438 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T06:02:15.438 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T06:02:15.438 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T06:02:15.438 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T06:02:15.972 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T06:02:15.972 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T06:02:15.972 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 4 )) 2024-09-16T06:02:15.972 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T06:02:15.973 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-16T06:02:16.172 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T06:02:16.173 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T06:02:16.173 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T06:02:16.173 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T06:02:16.173 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T06:02:16.173 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T06:02:16.173 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T06:02:16.173 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T06:02:16.538 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T06:02:16.538 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T06:02:16.538 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T06:02:16.538 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T06:02:17.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-16T06:02:17.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T06:02:17.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T06:02:17.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T06:02:17.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T06:02:17.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T06:02:17.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T06:02:17.041 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T06:02:17.041 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T06:02:17.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T06:02:17.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T06:02:17.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 4 )) 2024-09-16T06:02:17.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T06:02:17.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-16T06:02:17.945 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T06:02:17.945 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T06:02:17.945 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T06:02:17.945 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T06:02:17.945 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T06:02:17.945 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T06:02:17.945 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T06:02:17.945 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T06:02:18.310 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T06:02:18.310 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T06:02:18.310 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T06:02:18.310 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T06:02:18.824 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-16T06:02:18.824 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T06:02:18.824 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T06:02:18.824 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T06:02:18.824 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T06:02:18.824 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T06:02:18.824 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T06:02:18.824 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T06:02:18.824 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T06:02:19.330 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T06:02:19.331 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T06:02:19.331 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 4 )) 2024-09-16T06:02:19.331 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T06:02:19.331 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.3 2024-09-16T06:02:19.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T06:02:19.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T06:02:19.632 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T06:02:19.632 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T06:02:19.632 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T06:02:19.632 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T06:02:19.632 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T06:02:19.632 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T06:02:19.996 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T06:02:19.996 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T06:02:19.996 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T06:02:19.997 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T06:02:20.495 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-16T06:02:20.495 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T06:02:20.495 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T06:02:20.495 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T06:02:20.495 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T06:02:20.495 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T06:02:20.495 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T06:02:20.495 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T06:02:20.495 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T06:02:21.062 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T06:02:21.062 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T06:02:21.063 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 4 )) 2024-09-16T06:02:21.063 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-16T06:02:21.558 INFO:tasks.workunit.client.0.smithi178.stdout:{ 2024-09-16T06:02:21.559 INFO:tasks.workunit.client.0.smithi178.stdout: "cluster_fingerprint": "655a76ec-b2c4-42f5-8205-bb9e33651286", 2024-09-16T06:02:21.559 INFO:tasks.workunit.client.0.smithi178.stdout: "version": "19.1.1-326-g8921c8ef", 2024-09-16T06:02:21.559 INFO:tasks.workunit.client.0.smithi178.stdout: "commit": "8921c8ef947807b717db041d85f0761b7914d5fc", 2024-09-16T06:02:21.559 INFO:tasks.workunit.client.0.smithi178.stdout: "timestamp": "2024-09-16T06:02:21.557965+0000", 2024-09-16T06:02:21.559 INFO:tasks.workunit.client.0.smithi178.stdout: "tag": "", 2024-09-16T06:02:21.559 INFO:tasks.workunit.client.0.smithi178.stdout: "health": { 2024-09-16T06:02:21.559 INFO:tasks.workunit.client.0.smithi178.stdout: "status": "HEALTH_WARN", 2024-09-16T06:02:21.559 INFO:tasks.workunit.client.0.smithi178.stdout: "checks": { 2024-09-16T06:02:21.559 INFO:tasks.workunit.client.0.smithi178.stdout: "PG_DEGRADED": { 2024-09-16T06:02:21.559 INFO:tasks.workunit.client.0.smithi178.stdout: "severity": "HEALTH_WARN", 2024-09-16T06:02:21.559 INFO:tasks.workunit.client.0.smithi178.stdout: "summary": { 2024-09-16T06:02:21.559 INFO:tasks.workunit.client.0.smithi178.stdout: "message": "Degraded data redundancy: 2/4 objects degraded (50.000%), 2 pgs degraded", 2024-09-16T06:02:21.559 INFO:tasks.workunit.client.0.smithi178.stdout: "count": 2 2024-09-16T06:02:21.559 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.560 INFO:tasks.workunit.client.0.smithi178.stdout: "detail": [ 2024-09-16T06:02:21.560 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.560 INFO:tasks.workunit.client.0.smithi178.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0]" 2024-09-16T06:02:21.560 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.560 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.560 INFO:tasks.workunit.client.0.smithi178.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [0]" 2024-09-16T06:02:21.560 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:02:21.560 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:02:21.560 INFO:tasks.workunit.client.0.smithi178.stdout: "muted": false 2024-09-16T06:02:21.560 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.560 INFO:tasks.workunit.client.0.smithi178.stdout: "TOO_FEW_OSDS": { 2024-09-16T06:02:21.560 INFO:tasks.workunit.client.0.smithi178.stdout: "severity": "HEALTH_WARN", 2024-09-16T06:02:21.560 INFO:tasks.workunit.client.0.smithi178.stdout: "summary": { 2024-09-16T06:02:21.561 INFO:tasks.workunit.client.0.smithi178.stdout: "message": "OSD count 1 < osd_pool_default_size 3", 2024-09-16T06:02:21.561 INFO:tasks.workunit.client.0.smithi178.stdout: "count": 2 2024-09-16T06:02:21.561 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.561 INFO:tasks.workunit.client.0.smithi178.stdout: "detail": [], 2024-09-16T06:02:21.561 INFO:tasks.workunit.client.0.smithi178.stdout: "muted": false 2024-09-16T06:02:21.561 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:02:21.561 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.561 INFO:tasks.workunit.client.0.smithi178.stdout: "mutes": [] 2024-09-16T06:02:21.561 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.561 INFO:tasks.workunit.client.0.smithi178.stdout: "monmap_first_committed": 1, 2024-09-16T06:02:21.561 INFO:tasks.workunit.client.0.smithi178.stdout: "monmap_last_committed": 1, 2024-09-16T06:02:21.561 INFO:tasks.workunit.client.0.smithi178.stdout: "monmap": { 2024-09-16T06:02:21.561 INFO:tasks.workunit.client.0.smithi178.stdout: "epoch": 1, 2024-09-16T06:02:21.561 INFO:tasks.workunit.client.0.smithi178.stdout: "fsid": "e36fe3b7-e285-4199-9297-eafea05d044b", 2024-09-16T06:02:21.562 INFO:tasks.workunit.client.0.smithi178.stdout: "modified": "2024-09-16T06:01:35.226705Z", 2024-09-16T06:02:21.562 INFO:tasks.workunit.client.0.smithi178.stdout: "created": "2024-09-16T06:01:35.226705Z", 2024-09-16T06:02:21.562 INFO:tasks.workunit.client.0.smithi178.stdout: "min_mon_release": 19, 2024-09-16T06:02:21.562 INFO:tasks.workunit.client.0.smithi178.stdout: "min_mon_release_name": "squid", 2024-09-16T06:02:21.562 INFO:tasks.workunit.client.0.smithi178.stdout: "election_strategy": 1, 2024-09-16T06:02:21.562 INFO:tasks.workunit.client.0.smithi178.stdout: "disallowed_leaders: ": "", 2024-09-16T06:02:21.562 INFO:tasks.workunit.client.0.smithi178.stdout: "stretch_mode": false, 2024-09-16T06:02:21.562 INFO:tasks.workunit.client.0.smithi178.stdout: "tiebreaker_mon": "", 2024-09-16T06:02:21.562 INFO:tasks.workunit.client.0.smithi178.stdout: "removed_ranks: ": "", 2024-09-16T06:02:21.562 INFO:tasks.workunit.client.0.smithi178.stdout: "features": { 2024-09-16T06:02:21.562 INFO:tasks.workunit.client.0.smithi178.stdout: "persistent": [ 2024-09-16T06:02:21.562 INFO:tasks.workunit.client.0.smithi178.stdout: "kraken", 2024-09-16T06:02:21.562 INFO:tasks.workunit.client.0.smithi178.stdout: "luminous", 2024-09-16T06:02:21.562 INFO:tasks.workunit.client.0.smithi178.stdout: "mimic", 2024-09-16T06:02:21.562 INFO:tasks.workunit.client.0.smithi178.stdout: "osdmap-prune", 2024-09-16T06:02:21.563 INFO:tasks.workunit.client.0.smithi178.stdout: "nautilus", 2024-09-16T06:02:21.563 INFO:tasks.workunit.client.0.smithi178.stdout: "octopus", 2024-09-16T06:02:21.563 INFO:tasks.workunit.client.0.smithi178.stdout: "pacific", 2024-09-16T06:02:21.563 INFO:tasks.workunit.client.0.smithi178.stdout: "elector-pinging", 2024-09-16T06:02:21.563 INFO:tasks.workunit.client.0.smithi178.stdout: "quincy", 2024-09-16T06:02:21.563 INFO:tasks.workunit.client.0.smithi178.stdout: "reef", 2024-09-16T06:02:21.563 INFO:tasks.workunit.client.0.smithi178.stdout: "squid" 2024-09-16T06:02:21.563 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:02:21.563 INFO:tasks.workunit.client.0.smithi178.stdout: "optional": [] 2024-09-16T06:02:21.563 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.563 INFO:tasks.workunit.client.0.smithi178.stdout: "mons": [ 2024-09-16T06:02:21.563 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.563 INFO:tasks.workunit.client.0.smithi178.stdout: "rank": 0, 2024-09-16T06:02:21.563 INFO:tasks.workunit.client.0.smithi178.stdout: "name": "a", 2024-09-16T06:02:21.564 INFO:tasks.workunit.client.0.smithi178.stdout: "public_addrs": { 2024-09-16T06:02:21.564 INFO:tasks.workunit.client.0.smithi178.stdout: "addrvec": [ 2024-09-16T06:02:21.564 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.564 INFO:tasks.workunit.client.0.smithi178.stdout: "type": "v2", 2024-09-16T06:02:21.564 INFO:tasks.workunit.client.0.smithi178.stdout: "addr": "127.0.0.1:7109", 2024-09-16T06:02:21.564 INFO:tasks.workunit.client.0.smithi178.stdout: "nonce": 0 2024-09-16T06:02:21.564 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:02:21.564 INFO:tasks.workunit.client.0.smithi178.stdout: ] 2024-09-16T06:02:21.564 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.564 INFO:tasks.workunit.client.0.smithi178.stdout: "addr": "127.0.0.1:7109/0", 2024-09-16T06:02:21.564 INFO:tasks.workunit.client.0.smithi178.stdout: "public_addr": "127.0.0.1:7109/0", 2024-09-16T06:02:21.564 INFO:tasks.workunit.client.0.smithi178.stdout: "priority": 0, 2024-09-16T06:02:21.564 INFO:tasks.workunit.client.0.smithi178.stdout: "weight": 0, 2024-09-16T06:02:21.565 INFO:tasks.workunit.client.0.smithi178.stdout: "crush_location": "{}" 2024-09-16T06:02:21.565 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:02:21.565 INFO:tasks.workunit.client.0.smithi178.stdout: ] 2024-09-16T06:02:21.565 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.565 INFO:tasks.workunit.client.0.smithi178.stdout: "quorum": [ 2024-09-16T06:02:21.565 INFO:tasks.workunit.client.0.smithi178.stdout: 0 2024-09-16T06:02:21.565 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:02:21.565 INFO:tasks.workunit.client.0.smithi178.stdout: "osdmap": { 2024-09-16T06:02:21.565 INFO:tasks.workunit.client.0.smithi178.stdout: "epoch": 13, 2024-09-16T06:02:21.565 INFO:tasks.workunit.client.0.smithi178.stdout: "fsid": "e36fe3b7-e285-4199-9297-eafea05d044b", 2024-09-16T06:02:21.565 INFO:tasks.workunit.client.0.smithi178.stdout: "created": "2024-09-16T06:01:36.020724+0000", 2024-09-16T06:02:21.565 INFO:tasks.workunit.client.0.smithi178.stdout: "modified": "2024-09-16T06:02:10.953448+0000", 2024-09-16T06:02:21.565 INFO:tasks.workunit.client.0.smithi178.stdout: "last_up_change": "2024-09-16T06:02:08.082964+0000", 2024-09-16T06:02:21.566 INFO:tasks.workunit.client.0.smithi178.stdout: "last_in_change": "2024-09-16T06:01:36.719674+0000", 2024-09-16T06:02:21.566 INFO:tasks.workunit.client.0.smithi178.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T06:02:21.566 INFO:tasks.workunit.client.0.smithi178.stdout: "flags_num": 5799936, 2024-09-16T06:02:21.566 INFO:tasks.workunit.client.0.smithi178.stdout: "flags_set": [ 2024-09-16T06:02:21.566 INFO:tasks.workunit.client.0.smithi178.stdout: "pglog_hardlimit", 2024-09-16T06:02:21.566 INFO:tasks.workunit.client.0.smithi178.stdout: "purged_snapdirs", 2024-09-16T06:02:21.566 INFO:tasks.workunit.client.0.smithi178.stdout: "recovery_deletes", 2024-09-16T06:02:21.566 INFO:tasks.workunit.client.0.smithi178.stdout: "sortbitwise" 2024-09-16T06:02:21.566 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:02:21.566 INFO:tasks.workunit.client.0.smithi178.stdout: "crush_version": 3, 2024-09-16T06:02:21.566 INFO:tasks.workunit.client.0.smithi178.stdout: "full_ratio": 0.99000000953674316, 2024-09-16T06:02:21.566 INFO:tasks.workunit.client.0.smithi178.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-16T06:02:21.566 INFO:tasks.workunit.client.0.smithi178.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T06:02:21.567 INFO:tasks.workunit.client.0.smithi178.stdout: "cluster_snapshot": "", 2024-09-16T06:02:21.567 INFO:tasks.workunit.client.0.smithi178.stdout: "pool_max": 1, 2024-09-16T06:02:21.567 INFO:tasks.workunit.client.0.smithi178.stdout: "max_osd": 1, 2024-09-16T06:02:21.567 INFO:tasks.workunit.client.0.smithi178.stdout: "require_min_compat_client": "luminous", 2024-09-16T06:02:21.567 INFO:tasks.workunit.client.0.smithi178.stdout: "min_compat_client": "jewel", 2024-09-16T06:02:21.567 INFO:tasks.workunit.client.0.smithi178.stdout: "require_osd_release": "squid", 2024-09-16T06:02:21.567 INFO:tasks.workunit.client.0.smithi178.stdout: "allow_crimson": false, 2024-09-16T06:02:21.567 INFO:tasks.workunit.client.0.smithi178.stdout: "pools": [ 2024-09-16T06:02:21.567 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.567 INFO:tasks.workunit.client.0.smithi178.stdout: "pool": 1, 2024-09-16T06:02:21.567 INFO:tasks.workunit.client.0.smithi178.stdout: "pool_name": "rbd", 2024-09-16T06:02:21.567 INFO:tasks.workunit.client.0.smithi178.stdout: "create_time": "2024-09-16T06:02:09.106756+0000", 2024-09-16T06:02:21.567 INFO:tasks.workunit.client.0.smithi178.stdout: "flags": 8193, 2024-09-16T06:02:21.567 INFO:tasks.workunit.client.0.smithi178.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T06:02:21.568 INFO:tasks.workunit.client.0.smithi178.stdout: "type": 1, 2024-09-16T06:02:21.568 INFO:tasks.workunit.client.0.smithi178.stdout: "size": 2, 2024-09-16T06:02:21.568 INFO:tasks.workunit.client.0.smithi178.stdout: "min_size": 1, 2024-09-16T06:02:21.568 INFO:tasks.workunit.client.0.smithi178.stdout: "crush_rule": 0, 2024-09-16T06:02:21.568 INFO:tasks.workunit.client.0.smithi178.stdout: "peering_crush_bucket_count": 0, 2024-09-16T06:02:21.568 INFO:tasks.workunit.client.0.smithi178.stdout: "peering_crush_bucket_target": 0, 2024-09-16T06:02:21.568 INFO:tasks.workunit.client.0.smithi178.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T06:02:21.568 INFO:tasks.workunit.client.0.smithi178.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T06:02:21.568 INFO:tasks.workunit.client.0.smithi178.stdout: "is_stretch_pool": false, 2024-09-16T06:02:21.568 INFO:tasks.workunit.client.0.smithi178.stdout: "object_hash": 2, 2024-09-16T06:02:21.568 INFO:tasks.workunit.client.0.smithi178.stdout: "pg_autoscale_mode": "off", 2024-09-16T06:02:21.568 INFO:tasks.workunit.client.0.smithi178.stdout: "pg_num": 4, 2024-09-16T06:02:21.568 INFO:tasks.workunit.client.0.smithi178.stdout: "pg_placement_num": 4, 2024-09-16T06:02:21.569 INFO:tasks.workunit.client.0.smithi178.stdout: "pg_placement_num_target": 4, 2024-09-16T06:02:21.569 INFO:tasks.workunit.client.0.smithi178.stdout: "pg_num_target": 4, 2024-09-16T06:02:21.569 INFO:tasks.workunit.client.0.smithi178.stdout: "pg_num_pending": 4, 2024-09-16T06:02:21.569 INFO:tasks.workunit.client.0.smithi178.stdout: "last_pg_merge_meta": { 2024-09-16T06:02:21.569 INFO:tasks.workunit.client.0.smithi178.stdout: "source_pgid": "0.0", 2024-09-16T06:02:21.569 INFO:tasks.workunit.client.0.smithi178.stdout: "ready_epoch": 0, 2024-09-16T06:02:21.569 INFO:tasks.workunit.client.0.smithi178.stdout: "last_epoch_started": 0, 2024-09-16T06:02:21.569 INFO:tasks.workunit.client.0.smithi178.stdout: "last_epoch_clean": 0, 2024-09-16T06:02:21.569 INFO:tasks.workunit.client.0.smithi178.stdout: "source_version": "0'0", 2024-09-16T06:02:21.569 INFO:tasks.workunit.client.0.smithi178.stdout: "target_version": "0'0" 2024-09-16T06:02:21.569 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.569 INFO:tasks.workunit.client.0.smithi178.stdout: "last_change": "13", 2024-09-16T06:02:21.569 INFO:tasks.workunit.client.0.smithi178.stdout: "last_force_op_resend": "0", 2024-09-16T06:02:21.570 INFO:tasks.workunit.client.0.smithi178.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T06:02:21.570 INFO:tasks.workunit.client.0.smithi178.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T06:02:21.570 INFO:tasks.workunit.client.0.smithi178.stdout: "auid": 0, 2024-09-16T06:02:21.570 INFO:tasks.workunit.client.0.smithi178.stdout: "snap_mode": "selfmanaged", 2024-09-16T06:02:21.570 INFO:tasks.workunit.client.0.smithi178.stdout: "snap_seq": 2, 2024-09-16T06:02:21.570 INFO:tasks.workunit.client.0.smithi178.stdout: "snap_epoch": 13, 2024-09-16T06:02:21.570 INFO:tasks.workunit.client.0.smithi178.stdout: "pool_snaps": [], 2024-09-16T06:02:21.570 INFO:tasks.workunit.client.0.smithi178.stdout: "removed_snaps": "[]", 2024-09-16T06:02:21.570 INFO:tasks.workunit.client.0.smithi178.stdout: "quota_max_bytes": 0, 2024-09-16T06:02:21.570 INFO:tasks.workunit.client.0.smithi178.stdout: "quota_max_objects": 0, 2024-09-16T06:02:21.570 INFO:tasks.workunit.client.0.smithi178.stdout: "tiers": [], 2024-09-16T06:02:21.570 INFO:tasks.workunit.client.0.smithi178.stdout: "tier_of": -1, 2024-09-16T06:02:21.570 INFO:tasks.workunit.client.0.smithi178.stdout: "read_tier": -1, 2024-09-16T06:02:21.570 INFO:tasks.workunit.client.0.smithi178.stdout: "write_tier": -1, 2024-09-16T06:02:21.571 INFO:tasks.workunit.client.0.smithi178.stdout: "cache_mode": "none", 2024-09-16T06:02:21.571 INFO:tasks.workunit.client.0.smithi178.stdout: "target_max_bytes": 0, 2024-09-16T06:02:21.571 INFO:tasks.workunit.client.0.smithi178.stdout: "target_max_objects": 0, 2024-09-16T06:02:21.571 INFO:tasks.workunit.client.0.smithi178.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T06:02:21.571 INFO:tasks.workunit.client.0.smithi178.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T06:02:21.571 INFO:tasks.workunit.client.0.smithi178.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T06:02:21.571 INFO:tasks.workunit.client.0.smithi178.stdout: "cache_min_flush_age": 0, 2024-09-16T06:02:21.571 INFO:tasks.workunit.client.0.smithi178.stdout: "cache_min_evict_age": 0, 2024-09-16T06:02:21.571 INFO:tasks.workunit.client.0.smithi178.stdout: "erasure_code_profile": "", 2024-09-16T06:02:21.571 INFO:tasks.workunit.client.0.smithi178.stdout: "hit_set_params": { 2024-09-16T06:02:21.571 INFO:tasks.workunit.client.0.smithi178.stdout: "type": "none" 2024-09-16T06:02:21.571 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.571 INFO:tasks.workunit.client.0.smithi178.stdout: "hit_set_period": 0, 2024-09-16T06:02:21.572 INFO:tasks.workunit.client.0.smithi178.stdout: "hit_set_count": 0, 2024-09-16T06:02:21.572 INFO:tasks.workunit.client.0.smithi178.stdout: "use_gmt_hitset": true, 2024-09-16T06:02:21.572 INFO:tasks.workunit.client.0.smithi178.stdout: "min_read_recency_for_promote": 0, 2024-09-16T06:02:21.572 INFO:tasks.workunit.client.0.smithi178.stdout: "min_write_recency_for_promote": 0, 2024-09-16T06:02:21.572 INFO:tasks.workunit.client.0.smithi178.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T06:02:21.572 INFO:tasks.workunit.client.0.smithi178.stdout: "hit_set_search_last_n": 0, 2024-09-16T06:02:21.572 INFO:tasks.workunit.client.0.smithi178.stdout: "grade_table": [], 2024-09-16T06:02:21.572 INFO:tasks.workunit.client.0.smithi178.stdout: "stripe_width": 0, 2024-09-16T06:02:21.572 INFO:tasks.workunit.client.0.smithi178.stdout: "expected_num_objects": 0, 2024-09-16T06:02:21.572 INFO:tasks.workunit.client.0.smithi178.stdout: "fast_read": false, 2024-09-16T06:02:21.572 INFO:tasks.workunit.client.0.smithi178.stdout: "options": {}, 2024-09-16T06:02:21.572 INFO:tasks.workunit.client.0.smithi178.stdout: "application_metadata": { 2024-09-16T06:02:21.573 INFO:tasks.workunit.client.0.smithi178.stdout: "rbd": {} 2024-09-16T06:02:21.573 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.573 INFO:tasks.workunit.client.0.smithi178.stdout: "read_balance": { 2024-09-16T06:02:21.573 INFO:tasks.workunit.client.0.smithi178.stdout: "score_type": "Fair distribution", 2024-09-16T06:02:21.573 INFO:tasks.workunit.client.0.smithi178.stdout: "score_acting": 1, 2024-09-16T06:02:21.573 INFO:tasks.workunit.client.0.smithi178.stdout: "score_stable": 1, 2024-09-16T06:02:21.573 INFO:tasks.workunit.client.0.smithi178.stdout: "optimal_score": 1, 2024-09-16T06:02:21.573 INFO:tasks.workunit.client.0.smithi178.stdout: "raw_score_acting": 1, 2024-09-16T06:02:21.573 INFO:tasks.workunit.client.0.smithi178.stdout: "raw_score_stable": 1, 2024-09-16T06:02:21.573 INFO:tasks.workunit.client.0.smithi178.stdout: "primary_affinity_weighted": 1, 2024-09-16T06:02:21.573 INFO:tasks.workunit.client.0.smithi178.stdout: "average_primary_affinity": 1, 2024-09-16T06:02:21.573 INFO:tasks.workunit.client.0.smithi178.stdout: "average_primary_affinity_weighted": 1 2024-09-16T06:02:21.573 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:02:21.574 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:02:21.574 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:02:21.574 INFO:tasks.workunit.client.0.smithi178.stdout: "osds": [ 2024-09-16T06:02:21.574 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.574 INFO:tasks.workunit.client.0.smithi178.stdout: "osd": 0, 2024-09-16T06:02:21.574 INFO:tasks.workunit.client.0.smithi178.stdout: "uuid": "6189a740-f623-496b-81d5-8c76481ac225", 2024-09-16T06:02:21.574 INFO:tasks.workunit.client.0.smithi178.stdout: "up": 1, 2024-09-16T06:02:21.574 INFO:tasks.workunit.client.0.smithi178.stdout: "in": 1, 2024-09-16T06:02:21.574 INFO:tasks.workunit.client.0.smithi178.stdout: "weight": 1, 2024-09-16T06:02:21.574 INFO:tasks.workunit.client.0.smithi178.stdout: "primary_affinity": 1, 2024-09-16T06:02:21.574 INFO:tasks.workunit.client.0.smithi178.stdout: "last_clean_begin": 0, 2024-09-16T06:02:21.574 INFO:tasks.workunit.client.0.smithi178.stdout: "last_clean_end": 0, 2024-09-16T06:02:21.574 INFO:tasks.workunit.client.0.smithi178.stdout: "up_from": 5, 2024-09-16T06:02:21.574 INFO:tasks.workunit.client.0.smithi178.stdout: "up_thru": 7, 2024-09-16T06:02:21.574 INFO:tasks.workunit.client.0.smithi178.stdout: "down_at": 0, 2024-09-16T06:02:21.575 INFO:tasks.workunit.client.0.smithi178.stdout: "lost_at": 0, 2024-09-16T06:02:21.575 INFO:tasks.workunit.client.0.smithi178.stdout: "public_addrs": { 2024-09-16T06:02:21.575 INFO:tasks.workunit.client.0.smithi178.stdout: "addrvec": [ 2024-09-16T06:02:21.575 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.575 INFO:tasks.workunit.client.0.smithi178.stdout: "type": "v2", 2024-09-16T06:02:21.575 INFO:tasks.workunit.client.0.smithi178.stdout: "addr": "127.0.0.1:6800", 2024-09-16T06:02:21.575 INFO:tasks.workunit.client.0.smithi178.stdout: "nonce": 1284002052 2024-09-16T06:02:21.575 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.575 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.575 INFO:tasks.workunit.client.0.smithi178.stdout: "type": "v1", 2024-09-16T06:02:21.575 INFO:tasks.workunit.client.0.smithi178.stdout: "addr": "127.0.0.1:6801", 2024-09-16T06:02:21.575 INFO:tasks.workunit.client.0.smithi178.stdout: "nonce": 1284002052 2024-09-16T06:02:21.575 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:02:21.575 INFO:tasks.workunit.client.0.smithi178.stdout: ] 2024-09-16T06:02:21.576 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.576 INFO:tasks.workunit.client.0.smithi178.stdout: "cluster_addrs": { 2024-09-16T06:02:21.576 INFO:tasks.workunit.client.0.smithi178.stdout: "addrvec": [ 2024-09-16T06:02:21.576 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.576 INFO:tasks.workunit.client.0.smithi178.stdout: "type": "v2", 2024-09-16T06:02:21.576 INFO:tasks.workunit.client.0.smithi178.stdout: "addr": "127.0.0.1:6802", 2024-09-16T06:02:21.576 INFO:tasks.workunit.client.0.smithi178.stdout: "nonce": 1284002052 2024-09-16T06:02:21.576 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.576 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.576 INFO:tasks.workunit.client.0.smithi178.stdout: "type": "v1", 2024-09-16T06:02:21.576 INFO:tasks.workunit.client.0.smithi178.stdout: "addr": "127.0.0.1:6803", 2024-09-16T06:02:21.576 INFO:tasks.workunit.client.0.smithi178.stdout: "nonce": 1284002052 2024-09-16T06:02:21.576 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:02:21.576 INFO:tasks.workunit.client.0.smithi178.stdout: ] 2024-09-16T06:02:21.577 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.577 INFO:tasks.workunit.client.0.smithi178.stdout: "heartbeat_back_addrs": { 2024-09-16T06:02:21.577 INFO:tasks.workunit.client.0.smithi178.stdout: "addrvec": [ 2024-09-16T06:02:21.577 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.577 INFO:tasks.workunit.client.0.smithi178.stdout: "type": "v2", 2024-09-16T06:02:21.577 INFO:tasks.workunit.client.0.smithi178.stdout: "addr": "127.0.0.1:6806", 2024-09-16T06:02:21.577 INFO:tasks.workunit.client.0.smithi178.stdout: "nonce": 1284002052 2024-09-16T06:02:21.577 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.577 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.577 INFO:tasks.workunit.client.0.smithi178.stdout: "type": "v1", 2024-09-16T06:02:21.577 INFO:tasks.workunit.client.0.smithi178.stdout: "addr": "127.0.0.1:6807", 2024-09-16T06:02:21.577 INFO:tasks.workunit.client.0.smithi178.stdout: "nonce": 1284002052 2024-09-16T06:02:21.577 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:02:21.577 INFO:tasks.workunit.client.0.smithi178.stdout: ] 2024-09-16T06:02:21.578 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.578 INFO:tasks.workunit.client.0.smithi178.stdout: "heartbeat_front_addrs": { 2024-09-16T06:02:21.578 INFO:tasks.workunit.client.0.smithi178.stdout: "addrvec": [ 2024-09-16T06:02:21.578 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.578 INFO:tasks.workunit.client.0.smithi178.stdout: "type": "v2", 2024-09-16T06:02:21.578 INFO:tasks.workunit.client.0.smithi178.stdout: "addr": "127.0.0.1:6804", 2024-09-16T06:02:21.578 INFO:tasks.workunit.client.0.smithi178.stdout: "nonce": 1284002052 2024-09-16T06:02:21.578 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.578 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.578 INFO:tasks.workunit.client.0.smithi178.stdout: "type": "v1", 2024-09-16T06:02:21.578 INFO:tasks.workunit.client.0.smithi178.stdout: "addr": "127.0.0.1:6805", 2024-09-16T06:02:21.578 INFO:tasks.workunit.client.0.smithi178.stdout: "nonce": 1284002052 2024-09-16T06:02:21.578 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:02:21.579 INFO:tasks.workunit.client.0.smithi178.stdout: ] 2024-09-16T06:02:21.579 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.579 INFO:tasks.workunit.client.0.smithi178.stdout: "public_addr": "127.0.0.1:6801/1284002052", 2024-09-16T06:02:21.579 INFO:tasks.workunit.client.0.smithi178.stdout: "cluster_addr": "127.0.0.1:6803/1284002052", 2024-09-16T06:02:21.579 INFO:tasks.workunit.client.0.smithi178.stdout: "heartbeat_back_addr": "127.0.0.1:6807/1284002052", 2024-09-16T06:02:21.579 INFO:tasks.workunit.client.0.smithi178.stdout: "heartbeat_front_addr": "127.0.0.1:6805/1284002052", 2024-09-16T06:02:21.579 INFO:tasks.workunit.client.0.smithi178.stdout: "state": [ 2024-09-16T06:02:21.579 INFO:tasks.workunit.client.0.smithi178.stdout: "exists", 2024-09-16T06:02:21.579 INFO:tasks.workunit.client.0.smithi178.stdout: "up" 2024-09-16T06:02:21.579 INFO:tasks.workunit.client.0.smithi178.stdout: ] 2024-09-16T06:02:21.579 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:02:21.579 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:02:21.579 INFO:tasks.workunit.client.0.smithi178.stdout: "osd_xinfo": [ 2024-09-16T06:02:21.579 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.580 INFO:tasks.workunit.client.0.smithi178.stdout: "osd": 0, 2024-09-16T06:02:21.580 INFO:tasks.workunit.client.0.smithi178.stdout: "down_stamp": "0.000000", 2024-09-16T06:02:21.580 INFO:tasks.workunit.client.0.smithi178.stdout: "laggy_probability": 0, 2024-09-16T06:02:21.580 INFO:tasks.workunit.client.0.smithi178.stdout: "laggy_interval": 0, 2024-09-16T06:02:21.580 INFO:tasks.workunit.client.0.smithi178.stdout: "features": 4540701547738038271, 2024-09-16T06:02:21.580 INFO:tasks.workunit.client.0.smithi178.stdout: "old_weight": 0, 2024-09-16T06:02:21.580 INFO:tasks.workunit.client.0.smithi178.stdout: "last_purged_snaps_scrub": "2024-09-16T06:02:07.823078+0000", 2024-09-16T06:02:21.580 INFO:tasks.workunit.client.0.smithi178.stdout: "dead_epoch": 0 2024-09-16T06:02:21.580 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:02:21.580 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:02:21.580 INFO:tasks.workunit.client.0.smithi178.stdout: "pg_upmap": [], 2024-09-16T06:02:21.580 INFO:tasks.workunit.client.0.smithi178.stdout: "pg_upmap_items": [], 2024-09-16T06:02:21.580 INFO:tasks.workunit.client.0.smithi178.stdout: "pg_upmap_primaries": [], 2024-09-16T06:02:21.580 INFO:tasks.workunit.client.0.smithi178.stdout: "pg_temp": [], 2024-09-16T06:02:21.581 INFO:tasks.workunit.client.0.smithi178.stdout: "primary_temp": [], 2024-09-16T06:02:21.581 INFO:tasks.workunit.client.0.smithi178.stdout: "blocklist": {}, 2024-09-16T06:02:21.581 INFO:tasks.workunit.client.0.smithi178.stdout: "range_blocklist": {}, 2024-09-16T06:02:21.581 INFO:tasks.workunit.client.0.smithi178.stdout: "erasure_code_profiles": { 2024-09-16T06:02:21.581 INFO:tasks.workunit.client.0.smithi178.stdout: "default": { 2024-09-16T06:02:21.581 INFO:tasks.workunit.client.0.smithi178.stdout: "k": "2", 2024-09-16T06:02:21.581 INFO:tasks.workunit.client.0.smithi178.stdout: "m": "2", 2024-09-16T06:02:21.581 INFO:tasks.workunit.client.0.smithi178.stdout: "plugin": "jerasure", 2024-09-16T06:02:21.581 INFO:tasks.workunit.client.0.smithi178.stdout: "technique": "reed_sol_van" 2024-09-16T06:02:21.581 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:02:21.581 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.581 INFO:tasks.workunit.client.0.smithi178.stdout: "removed_snaps_queue": [ 2024-09-16T06:02:21.581 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.581 INFO:tasks.workunit.client.0.smithi178.stdout: "pool": 1, 2024-09-16T06:02:21.582 INFO:tasks.workunit.client.0.smithi178.stdout: "snaps": [ 2024-09-16T06:02:21.582 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.582 INFO:tasks.workunit.client.0.smithi178.stdout: "begin": 2, 2024-09-16T06:02:21.582 INFO:tasks.workunit.client.0.smithi178.stdout: "length": 1 2024-09-16T06:02:21.582 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:02:21.582 INFO:tasks.workunit.client.0.smithi178.stdout: ] 2024-09-16T06:02:21.582 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:02:21.582 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:02:21.582 INFO:tasks.workunit.client.0.smithi178.stdout: "new_removed_snaps": [ 2024-09-16T06:02:21.582 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.582 INFO:tasks.workunit.client.0.smithi178.stdout: "pool": 1, 2024-09-16T06:02:21.582 INFO:tasks.workunit.client.0.smithi178.stdout: "snaps": [ 2024-09-16T06:02:21.582 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.583 INFO:tasks.workunit.client.0.smithi178.stdout: "begin": 2, 2024-09-16T06:02:21.583 INFO:tasks.workunit.client.0.smithi178.stdout: "length": 1 2024-09-16T06:02:21.583 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:02:21.583 INFO:tasks.workunit.client.0.smithi178.stdout: ] 2024-09-16T06:02:21.583 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:02:21.583 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:02:21.583 INFO:tasks.workunit.client.0.smithi178.stdout: "new_purged_snaps": [], 2024-09-16T06:02:21.583 INFO:tasks.workunit.client.0.smithi178.stdout: "crush_node_flags": {}, 2024-09-16T06:02:21.583 INFO:tasks.workunit.client.0.smithi178.stdout: "device_class_flags": {}, 2024-09-16T06:02:21.583 INFO:tasks.workunit.client.0.smithi178.stdout: "stretch_mode": { 2024-09-16T06:02:21.583 INFO:tasks.workunit.client.0.smithi178.stdout: "stretch_mode_enabled": false, 2024-09-16T06:02:21.584 INFO:tasks.workunit.client.0.smithi178.stdout: "stretch_bucket_count": 0, 2024-09-16T06:02:21.584 INFO:tasks.workunit.client.0.smithi178.stdout: "degraded_stretch_mode": 0, 2024-09-16T06:02:21.584 INFO:tasks.workunit.client.0.smithi178.stdout: "recovering_stretch_mode": 0, 2024-09-16T06:02:21.584 INFO:tasks.workunit.client.0.smithi178.stdout: "stretch_mode_bucket": 0 2024-09-16T06:02:21.584 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:02:21.584 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.584 INFO:tasks.workunit.client.0.smithi178.stdout: "osd_metadata": [ 2024-09-16T06:02:21.584 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.584 INFO:tasks.workunit.client.0.smithi178.stdout: "id": 0, 2024-09-16T06:02:21.584 INFO:tasks.workunit.client.0.smithi178.stdout: "arch": "x86_64", 2024-09-16T06:02:21.584 INFO:tasks.workunit.client.0.smithi178.stdout: "back_addr": "[v2:127.0.0.1:6802/1284002052,v1:127.0.0.1:6803/1284002052]", 2024-09-16T06:02:21.584 INFO:tasks.workunit.client.0.smithi178.stdout: "back_iface": "", 2024-09-16T06:02:21.584 INFO:tasks.workunit.client.0.smithi178.stdout: "bluefs": "1", 2024-09-16T06:02:21.585 INFO:tasks.workunit.client.0.smithi178.stdout: "bluefs_dedicated_db": "0", 2024-09-16T06:02:21.585 INFO:tasks.workunit.client.0.smithi178.stdout: "bluefs_dedicated_wal": "0", 2024-09-16T06:02:21.585 INFO:tasks.workunit.client.0.smithi178.stdout: "bluefs_single_shared_device": "1", 2024-09-16T06:02:21.585 INFO:tasks.workunit.client.0.smithi178.stdout: "bluestore_allocation_from_file": "0", 2024-09-16T06:02:21.585 INFO:tasks.workunit.client.0.smithi178.stdout: "bluestore_bdev_access_mode": "file", 2024-09-16T06:02:21.585 INFO:tasks.workunit.client.0.smithi178.stdout: "bluestore_bdev_block_size": "4096", 2024-09-16T06:02:21.585 INFO:tasks.workunit.client.0.smithi178.stdout: "bluestore_bdev_devices": "sda", 2024-09-16T06:02:21.585 INFO:tasks.workunit.client.0.smithi178.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-16T06:02:21.585 INFO:tasks.workunit.client.0.smithi178.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-16T06:02:21.585 INFO:tasks.workunit.client.0.smithi178.stdout: "bluestore_bdev_path": "td/ceph-helpers/0/block", 2024-09-16T06:02:21.585 INFO:tasks.workunit.client.0.smithi178.stdout: "bluestore_bdev_rotational": "1", 2024-09-16T06:02:21.586 INFO:tasks.workunit.client.0.smithi178.stdout: "bluestore_bdev_size": "107374182400", 2024-09-16T06:02:21.586 INFO:tasks.workunit.client.0.smithi178.stdout: "bluestore_bdev_support_discard": "1", 2024-09-16T06:02:21.586 INFO:tasks.workunit.client.0.smithi178.stdout: "bluestore_bdev_type": "hdd", 2024-09-16T06:02:21.586 INFO:tasks.workunit.client.0.smithi178.stdout: "bluestore_min_alloc_size": "4096", 2024-09-16T06:02:21.586 INFO:tasks.workunit.client.0.smithi178.stdout: "ceph_release": "squid", 2024-09-16T06:02:21.586 INFO:tasks.workunit.client.0.smithi178.stdout: "ceph_version": "ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable)", 2024-09-16T06:02:21.586 INFO:tasks.workunit.client.0.smithi178.stdout: "ceph_version_short": "19.1.1-326-g8921c8ef", 2024-09-16T06:02:21.586 INFO:tasks.workunit.client.0.smithi178.stdout: "ceph_version_when_created": "ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable)", 2024-09-16T06:02:21.586 INFO:tasks.workunit.client.0.smithi178.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-16T06:02:21.587 INFO:tasks.workunit.client.0.smithi178.stdout: "created_at": "2024-09-16T06:01:41.729915Z", 2024-09-16T06:02:21.587 INFO:tasks.workunit.client.0.smithi178.stdout: "default_device_class": "hdd", 2024-09-16T06:02:21.587 INFO:tasks.workunit.client.0.smithi178.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XFJ4", 2024-09-16T06:02:21.587 INFO:tasks.workunit.client.0.smithi178.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-16T06:02:21.587 INFO:tasks.workunit.client.0.smithi178.stdout: "devices": "sda", 2024-09-16T06:02:21.587 INFO:tasks.workunit.client.0.smithi178.stdout: "distro": "centos", 2024-09-16T06:02:21.587 INFO:tasks.workunit.client.0.smithi178.stdout: "distro_description": "CentOS Stream 9", 2024-09-16T06:02:21.587 INFO:tasks.workunit.client.0.smithi178.stdout: "distro_version": "9", 2024-09-16T06:02:21.587 INFO:tasks.workunit.client.0.smithi178.stdout: "front_addr": "[v2:127.0.0.1:6800/1284002052,v1:127.0.0.1:6801/1284002052]", 2024-09-16T06:02:21.587 INFO:tasks.workunit.client.0.smithi178.stdout: "front_iface": "", 2024-09-16T06:02:21.587 INFO:tasks.workunit.client.0.smithi178.stdout: "hb_back_addr": "[v2:127.0.0.1:6806/1284002052,v1:127.0.0.1:6807/1284002052]", 2024-09-16T06:02:21.587 INFO:tasks.workunit.client.0.smithi178.stdout: "hb_front_addr": "[v2:127.0.0.1:6804/1284002052,v1:127.0.0.1:6805/1284002052]", 2024-09-16T06:02:21.587 INFO:tasks.workunit.client.0.smithi178.stdout: "hostname": "smithi178", 2024-09-16T06:02:21.588 INFO:tasks.workunit.client.0.smithi178.stdout: "journal_rotational": "1", 2024-09-16T06:02:21.588 INFO:tasks.workunit.client.0.smithi178.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-16T06:02:21.588 INFO:tasks.workunit.client.0.smithi178.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-16T06:02:21.588 INFO:tasks.workunit.client.0.smithi178.stdout: "mem_swap_kb": "0", 2024-09-16T06:02:21.588 INFO:tasks.workunit.client.0.smithi178.stdout: "mem_total_kb": "32487132", 2024-09-16T06:02:21.588 INFO:tasks.workunit.client.0.smithi178.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-16T06:02:21.588 INFO:tasks.workunit.client.0.smithi178.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-16T06:02:21.588 INFO:tasks.workunit.client.0.smithi178.stdout: "os": "Linux", 2024-09-16T06:02:21.588 INFO:tasks.workunit.client.0.smithi178.stdout: "osd_data": "td/ceph-helpers/0", 2024-09-16T06:02:21.588 INFO:tasks.workunit.client.0.smithi178.stdout: "osd_objectstore": "bluestore", 2024-09-16T06:02:21.588 INFO:tasks.workunit.client.0.smithi178.stdout: "osdspec_affinity": "", 2024-09-16T06:02:21.588 INFO:tasks.workunit.client.0.smithi178.stdout: "rotational": "1" 2024-09-16T06:02:21.588 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:02:21.589 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:02:21.589 INFO:tasks.workunit.client.0.smithi178.stdout: "osdmap_clean_epochs": { 2024-09-16T06:02:21.589 INFO:tasks.workunit.client.0.smithi178.stdout: "min_last_epoch_clean": 0, 2024-09-16T06:02:21.589 INFO:tasks.workunit.client.0.smithi178.stdout: "last_epoch_clean": { 2024-09-16T06:02:21.589 INFO:tasks.workunit.client.0.smithi178.stdout: "per_pool": [] 2024-09-16T06:02:21.589 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.589 INFO:tasks.workunit.client.0.smithi178.stdout: "osd_epochs": [ 2024-09-16T06:02:21.589 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.589 INFO:tasks.workunit.client.0.smithi178.stdout: "id": 0, 2024-09-16T06:02:21.589 INFO:tasks.workunit.client.0.smithi178.stdout: "epoch": 5 2024-09-16T06:02:21.589 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:02:21.589 INFO:tasks.workunit.client.0.smithi178.stdout: ] 2024-09-16T06:02:21.589 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.589 INFO:tasks.workunit.client.0.smithi178.stdout: "osdmap_first_committed": 1, 2024-09-16T06:02:21.590 INFO:tasks.workunit.client.0.smithi178.stdout: "osdmap_last_committed": 13, 2024-09-16T06:02:21.590 INFO:tasks.workunit.client.0.smithi178.stdout: "crushmap": { 2024-09-16T06:02:21.590 INFO:tasks.workunit.client.0.smithi178.stdout: "devices": [ 2024-09-16T06:02:21.590 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.590 INFO:tasks.workunit.client.0.smithi178.stdout: "id": 0, 2024-09-16T06:02:21.590 INFO:tasks.workunit.client.0.smithi178.stdout: "name": "osd.0", 2024-09-16T06:02:21.590 INFO:tasks.workunit.client.0.smithi178.stdout: "class": "hdd" 2024-09-16T06:02:21.590 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:02:21.590 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:02:21.590 INFO:tasks.workunit.client.0.smithi178.stdout: "types": [ 2024-09-16T06:02:21.590 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.590 INFO:tasks.workunit.client.0.smithi178.stdout: "type_id": 0, 2024-09-16T06:02:21.590 INFO:tasks.workunit.client.0.smithi178.stdout: "name": "osd" 2024-09-16T06:02:21.590 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.590 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.591 INFO:tasks.workunit.client.0.smithi178.stdout: "type_id": 1, 2024-09-16T06:02:21.591 INFO:tasks.workunit.client.0.smithi178.stdout: "name": "host" 2024-09-16T06:02:21.591 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.591 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.591 INFO:tasks.workunit.client.0.smithi178.stdout: "type_id": 2, 2024-09-16T06:02:21.591 INFO:tasks.workunit.client.0.smithi178.stdout: "name": "chassis" 2024-09-16T06:02:21.591 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.591 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.591 INFO:tasks.workunit.client.0.smithi178.stdout: "type_id": 3, 2024-09-16T06:02:21.591 INFO:tasks.workunit.client.0.smithi178.stdout: "name": "rack" 2024-09-16T06:02:21.591 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.591 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.591 INFO:tasks.workunit.client.0.smithi178.stdout: "type_id": 4, 2024-09-16T06:02:21.591 INFO:tasks.workunit.client.0.smithi178.stdout: "name": "row" 2024-09-16T06:02:21.592 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.592 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.592 INFO:tasks.workunit.client.0.smithi178.stdout: "type_id": 5, 2024-09-16T06:02:21.592 INFO:tasks.workunit.client.0.smithi178.stdout: "name": "pdu" 2024-09-16T06:02:21.592 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.592 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.592 INFO:tasks.workunit.client.0.smithi178.stdout: "type_id": 6, 2024-09-16T06:02:21.592 INFO:tasks.workunit.client.0.smithi178.stdout: "name": "pod" 2024-09-16T06:02:21.592 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.592 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.592 INFO:tasks.workunit.client.0.smithi178.stdout: "type_id": 7, 2024-09-16T06:02:21.592 INFO:tasks.workunit.client.0.smithi178.stdout: "name": "room" 2024-09-16T06:02:21.592 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.592 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.593 INFO:tasks.workunit.client.0.smithi178.stdout: "type_id": 8, 2024-09-16T06:02:21.593 INFO:tasks.workunit.client.0.smithi178.stdout: "name": "datacenter" 2024-09-16T06:02:21.593 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.593 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.593 INFO:tasks.workunit.client.0.smithi178.stdout: "type_id": 9, 2024-09-16T06:02:21.593 INFO:tasks.workunit.client.0.smithi178.stdout: "name": "zone" 2024-09-16T06:02:21.593 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.593 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.593 INFO:tasks.workunit.client.0.smithi178.stdout: "type_id": 10, 2024-09-16T06:02:21.593 INFO:tasks.workunit.client.0.smithi178.stdout: "name": "region" 2024-09-16T06:02:21.593 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.593 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.593 INFO:tasks.workunit.client.0.smithi178.stdout: "type_id": 11, 2024-09-16T06:02:21.593 INFO:tasks.workunit.client.0.smithi178.stdout: "name": "root" 2024-09-16T06:02:21.594 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:02:21.594 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:02:21.594 INFO:tasks.workunit.client.0.smithi178.stdout: "buckets": [ 2024-09-16T06:02:21.594 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.594 INFO:tasks.workunit.client.0.smithi178.stdout: "id": -1, 2024-09-16T06:02:21.594 INFO:tasks.workunit.client.0.smithi178.stdout: "name": "default", 2024-09-16T06:02:21.594 INFO:tasks.workunit.client.0.smithi178.stdout: "type_id": 11, 2024-09-16T06:02:21.594 INFO:tasks.workunit.client.0.smithi178.stdout: "type_name": "root", 2024-09-16T06:02:21.594 INFO:tasks.workunit.client.0.smithi178.stdout: "weight": 6402, 2024-09-16T06:02:21.594 INFO:tasks.workunit.client.0.smithi178.stdout: "alg": "straw2", 2024-09-16T06:02:21.594 INFO:tasks.workunit.client.0.smithi178.stdout: "hash": "rjenkins1", 2024-09-16T06:02:21.594 INFO:tasks.workunit.client.0.smithi178.stdout: "items": [ 2024-09-16T06:02:21.594 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.594 INFO:tasks.workunit.client.0.smithi178.stdout: "id": -3, 2024-09-16T06:02:21.595 INFO:tasks.workunit.client.0.smithi178.stdout: "weight": 6402, 2024-09-16T06:02:21.595 INFO:tasks.workunit.client.0.smithi178.stdout: "pos": 0 2024-09-16T06:02:21.595 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:02:21.595 INFO:tasks.workunit.client.0.smithi178.stdout: ] 2024-09-16T06:02:21.595 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.595 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.595 INFO:tasks.workunit.client.0.smithi178.stdout: "id": -2, 2024-09-16T06:02:21.595 INFO:tasks.workunit.client.0.smithi178.stdout: "name": "default~hdd", 2024-09-16T06:02:21.595 INFO:tasks.workunit.client.0.smithi178.stdout: "type_id": 11, 2024-09-16T06:02:21.595 INFO:tasks.workunit.client.0.smithi178.stdout: "type_name": "root", 2024-09-16T06:02:21.595 INFO:tasks.workunit.client.0.smithi178.stdout: "weight": 6402, 2024-09-16T06:02:21.595 INFO:tasks.workunit.client.0.smithi178.stdout: "alg": "straw2", 2024-09-16T06:02:21.595 INFO:tasks.workunit.client.0.smithi178.stdout: "hash": "rjenkins1", 2024-09-16T06:02:21.595 INFO:tasks.workunit.client.0.smithi178.stdout: "items": [ 2024-09-16T06:02:21.596 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.596 INFO:tasks.workunit.client.0.smithi178.stdout: "id": -4, 2024-09-16T06:02:21.596 INFO:tasks.workunit.client.0.smithi178.stdout: "weight": 6402, 2024-09-16T06:02:21.596 INFO:tasks.workunit.client.0.smithi178.stdout: "pos": 0 2024-09-16T06:02:21.596 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:02:21.596 INFO:tasks.workunit.client.0.smithi178.stdout: ] 2024-09-16T06:02:21.596 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.596 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.596 INFO:tasks.workunit.client.0.smithi178.stdout: "id": -3, 2024-09-16T06:02:21.596 INFO:tasks.workunit.client.0.smithi178.stdout: "name": "smithi178", 2024-09-16T06:02:21.596 INFO:tasks.workunit.client.0.smithi178.stdout: "type_id": 1, 2024-09-16T06:02:21.596 INFO:tasks.workunit.client.0.smithi178.stdout: "type_name": "host", 2024-09-16T06:02:21.596 INFO:tasks.workunit.client.0.smithi178.stdout: "weight": 6402, 2024-09-16T06:02:21.597 INFO:tasks.workunit.client.0.smithi178.stdout: "alg": "straw2", 2024-09-16T06:02:21.597 INFO:tasks.workunit.client.0.smithi178.stdout: "hash": "rjenkins1", 2024-09-16T06:02:21.597 INFO:tasks.workunit.client.0.smithi178.stdout: "items": [ 2024-09-16T06:02:21.597 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.597 INFO:tasks.workunit.client.0.smithi178.stdout: "id": 0, 2024-09-16T06:02:21.597 INFO:tasks.workunit.client.0.smithi178.stdout: "weight": 6402, 2024-09-16T06:02:21.597 INFO:tasks.workunit.client.0.smithi178.stdout: "pos": 0 2024-09-16T06:02:21.597 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:02:21.597 INFO:tasks.workunit.client.0.smithi178.stdout: ] 2024-09-16T06:02:21.597 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.597 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.597 INFO:tasks.workunit.client.0.smithi178.stdout: "id": -4, 2024-09-16T06:02:21.597 INFO:tasks.workunit.client.0.smithi178.stdout: "name": "smithi178~hdd", 2024-09-16T06:02:21.597 INFO:tasks.workunit.client.0.smithi178.stdout: "type_id": 1, 2024-09-16T06:02:21.598 INFO:tasks.workunit.client.0.smithi178.stdout: "type_name": "host", 2024-09-16T06:02:21.598 INFO:tasks.workunit.client.0.smithi178.stdout: "weight": 6402, 2024-09-16T06:02:21.598 INFO:tasks.workunit.client.0.smithi178.stdout: "alg": "straw2", 2024-09-16T06:02:21.598 INFO:tasks.workunit.client.0.smithi178.stdout: "hash": "rjenkins1", 2024-09-16T06:02:21.598 INFO:tasks.workunit.client.0.smithi178.stdout: "items": [ 2024-09-16T06:02:21.598 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.598 INFO:tasks.workunit.client.0.smithi178.stdout: "id": 0, 2024-09-16T06:02:21.598 INFO:tasks.workunit.client.0.smithi178.stdout: "weight": 6402, 2024-09-16T06:02:21.598 INFO:tasks.workunit.client.0.smithi178.stdout: "pos": 0 2024-09-16T06:02:21.598 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:02:21.598 INFO:tasks.workunit.client.0.smithi178.stdout: ] 2024-09-16T06:02:21.598 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:02:21.598 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:02:21.599 INFO:tasks.workunit.client.0.smithi178.stdout: "rules": [ 2024-09-16T06:02:21.599 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.599 INFO:tasks.workunit.client.0.smithi178.stdout: "rule_id": 0, 2024-09-16T06:02:21.599 INFO:tasks.workunit.client.0.smithi178.stdout: "rule_name": "replicated_rule", 2024-09-16T06:02:21.599 INFO:tasks.workunit.client.0.smithi178.stdout: "type": 1, 2024-09-16T06:02:21.599 INFO:tasks.workunit.client.0.smithi178.stdout: "steps": [ 2024-09-16T06:02:21.599 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.599 INFO:tasks.workunit.client.0.smithi178.stdout: "op": "take", 2024-09-16T06:02:21.599 INFO:tasks.workunit.client.0.smithi178.stdout: "item": -1, 2024-09-16T06:02:21.599 INFO:tasks.workunit.client.0.smithi178.stdout: "item_name": "default" 2024-09-16T06:02:21.599 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.599 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.599 INFO:tasks.workunit.client.0.smithi178.stdout: "op": "choose_firstn", 2024-09-16T06:02:21.599 INFO:tasks.workunit.client.0.smithi178.stdout: "num": 0, 2024-09-16T06:02:21.600 INFO:tasks.workunit.client.0.smithi178.stdout: "type": "osd" 2024-09-16T06:02:21.600 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.600 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.600 INFO:tasks.workunit.client.0.smithi178.stdout: "op": "emit" 2024-09-16T06:02:21.600 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:02:21.600 INFO:tasks.workunit.client.0.smithi178.stdout: ] 2024-09-16T06:02:21.600 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:02:21.600 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:02:21.600 INFO:tasks.workunit.client.0.smithi178.stdout: "tunables": { 2024-09-16T06:02:21.600 INFO:tasks.workunit.client.0.smithi178.stdout: "choose_local_tries": 0, 2024-09-16T06:02:21.600 INFO:tasks.workunit.client.0.smithi178.stdout: "choose_local_fallback_tries": 0, 2024-09-16T06:02:21.600 INFO:tasks.workunit.client.0.smithi178.stdout: "choose_total_tries": 50, 2024-09-16T06:02:21.600 INFO:tasks.workunit.client.0.smithi178.stdout: "chooseleaf_descend_once": 1, 2024-09-16T06:02:21.600 INFO:tasks.workunit.client.0.smithi178.stdout: "chooseleaf_vary_r": 1, 2024-09-16T06:02:21.601 INFO:tasks.workunit.client.0.smithi178.stdout: "chooseleaf_stable": 1, 2024-09-16T06:02:21.601 INFO:tasks.workunit.client.0.smithi178.stdout: "msr_descents": 100, 2024-09-16T06:02:21.601 INFO:tasks.workunit.client.0.smithi178.stdout: "msr_collision_tries": 100, 2024-09-16T06:02:21.601 INFO:tasks.workunit.client.0.smithi178.stdout: "straw_calc_version": 1, 2024-09-16T06:02:21.601 INFO:tasks.workunit.client.0.smithi178.stdout: "allowed_bucket_algs": 54, 2024-09-16T06:02:21.601 INFO:tasks.workunit.client.0.smithi178.stdout: "profile": "jewel", 2024-09-16T06:02:21.601 INFO:tasks.workunit.client.0.smithi178.stdout: "optimal_tunables": 1, 2024-09-16T06:02:21.601 INFO:tasks.workunit.client.0.smithi178.stdout: "legacy_tunables": 0, 2024-09-16T06:02:21.601 INFO:tasks.workunit.client.0.smithi178.stdout: "minimum_required_version": "jewel", 2024-09-16T06:02:21.601 INFO:tasks.workunit.client.0.smithi178.stdout: "require_feature_tunables": 1, 2024-09-16T06:02:21.601 INFO:tasks.workunit.client.0.smithi178.stdout: "require_feature_tunables2": 1, 2024-09-16T06:02:21.601 INFO:tasks.workunit.client.0.smithi178.stdout: "has_v2_rules": 0, 2024-09-16T06:02:21.601 INFO:tasks.workunit.client.0.smithi178.stdout: "require_feature_tunables3": 1, 2024-09-16T06:02:21.602 INFO:tasks.workunit.client.0.smithi178.stdout: "has_v3_rules": 0, 2024-09-16T06:02:21.602 INFO:tasks.workunit.client.0.smithi178.stdout: "has_v4_buckets": 1, 2024-09-16T06:02:21.602 INFO:tasks.workunit.client.0.smithi178.stdout: "require_feature_tunables5": 1, 2024-09-16T06:02:21.602 INFO:tasks.workunit.client.0.smithi178.stdout: "has_v5_rules": 0, 2024-09-16T06:02:21.602 INFO:tasks.workunit.client.0.smithi178.stdout: "has_msr_rules": 0 2024-09-16T06:02:21.602 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.602 INFO:tasks.workunit.client.0.smithi178.stdout: "choose_args": {} 2024-09-16T06:02:21.602 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.602 INFO:tasks.workunit.client.0.smithi178.stdout: "fsmap": { 2024-09-16T06:02:21.602 INFO:tasks.workunit.client.0.smithi178.stdout: "epoch": 1, 2024-09-16T06:02:21.602 INFO:tasks.workunit.client.0.smithi178.stdout: "btime": "2024-09-16T06:01:35:996071+0000", 2024-09-16T06:02:21.602 INFO:tasks.workunit.client.0.smithi178.stdout: "default_fscid": -1, 2024-09-16T06:02:21.602 INFO:tasks.workunit.client.0.smithi178.stdout: "compat": { 2024-09-16T06:02:21.602 INFO:tasks.workunit.client.0.smithi178.stdout: "compat": {}, 2024-09-16T06:02:21.603 INFO:tasks.workunit.client.0.smithi178.stdout: "ro_compat": {}, 2024-09-16T06:02:21.603 INFO:tasks.workunit.client.0.smithi178.stdout: "incompat": { 2024-09-16T06:02:21.603 INFO:tasks.workunit.client.0.smithi178.stdout: "feature_1": "base v0.20", 2024-09-16T06:02:21.603 INFO:tasks.workunit.client.0.smithi178.stdout: "feature_2": "client writeable ranges", 2024-09-16T06:02:21.603 INFO:tasks.workunit.client.0.smithi178.stdout: "feature_3": "default file layouts on dirs", 2024-09-16T06:02:21.603 INFO:tasks.workunit.client.0.smithi178.stdout: "feature_4": "dir inode in separate object", 2024-09-16T06:02:21.603 INFO:tasks.workunit.client.0.smithi178.stdout: "feature_5": "mds uses versioned encoding", 2024-09-16T06:02:21.603 INFO:tasks.workunit.client.0.smithi178.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-16T06:02:21.603 INFO:tasks.workunit.client.0.smithi178.stdout: "feature_8": "no anchor table", 2024-09-16T06:02:21.603 INFO:tasks.workunit.client.0.smithi178.stdout: "feature_9": "file layout v2", 2024-09-16T06:02:21.603 INFO:tasks.workunit.client.0.smithi178.stdout: "feature_10": "snaprealm v2", 2024-09-16T06:02:21.603 INFO:tasks.workunit.client.0.smithi178.stdout: "feature_11": "minor log segments", 2024-09-16T06:02:21.603 INFO:tasks.workunit.client.0.smithi178.stdout: "feature_12": "quiesce subvolumes" 2024-09-16T06:02:21.603 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:02:21.604 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.604 INFO:tasks.workunit.client.0.smithi178.stdout: "feature_flags": { 2024-09-16T06:02:21.604 INFO:tasks.workunit.client.0.smithi178.stdout: "enable_multiple": true, 2024-09-16T06:02:21.604 INFO:tasks.workunit.client.0.smithi178.stdout: "ever_enabled_multiple": true 2024-09-16T06:02:21.604 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.604 INFO:tasks.workunit.client.0.smithi178.stdout: "standbys": [], 2024-09-16T06:02:21.604 INFO:tasks.workunit.client.0.smithi178.stdout: "filesystems": [] 2024-09-16T06:02:21.604 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.604 INFO:tasks.workunit.client.0.smithi178.stdout: "mdsmap_first_committed": 1, 2024-09-16T06:02:21.604 INFO:tasks.workunit.client.0.smithi178.stdout: "mdsmap_last_committed": 1, 2024-09-16T06:02:21.604 INFO:tasks.workunit.client.0.smithi178.stdout: "auth": { 2024-09-16T06:02:21.604 INFO:tasks.workunit.client.0.smithi178.stdout: "first_committed": 1, 2024-09-16T06:02:21.604 INFO:tasks.workunit.client.0.smithi178.stdout: "last_committed": 3, 2024-09-16T06:02:21.604 INFO:tasks.workunit.client.0.smithi178.stdout: "num_secrets": 1 2024-09-16T06:02:21.605 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.605 INFO:tasks.workunit.client.0.smithi178.stdout: "num_pg": 4, 2024-09-16T06:02:21.605 INFO:tasks.workunit.client.0.smithi178.stdout: "num_pg_active": 4, 2024-09-16T06:02:21.605 INFO:tasks.workunit.client.0.smithi178.stdout: "num_pg_unknown": 0, 2024-09-16T06:02:21.605 INFO:tasks.workunit.client.0.smithi178.stdout: "num_osd": 1, 2024-09-16T06:02:21.605 INFO:tasks.workunit.client.0.smithi178.stdout: "pool_sum": { 2024-09-16T06:02:21.605 INFO:tasks.workunit.client.0.smithi178.stdout: "stat_sum": { 2024-09-16T06:02:21.605 INFO:tasks.workunit.client.0.smithi178.stdout: "num_bytes": 19, 2024-09-16T06:02:21.605 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects": 2, 2024-09-16T06:02:21.605 INFO:tasks.workunit.client.0.smithi178.stdout: "num_object_clones": 0, 2024-09-16T06:02:21.605 INFO:tasks.workunit.client.0.smithi178.stdout: "num_object_copies": 4, 2024-09-16T06:02:21.605 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_missing_on_primary": 0, 2024-09-16T06:02:21.605 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_missing": 0, 2024-09-16T06:02:21.605 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_degraded": 2, 2024-09-16T06:02:21.605 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_misplaced": 0, 2024-09-16T06:02:21.606 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_unfound": 0, 2024-09-16T06:02:21.606 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_dirty": 2, 2024-09-16T06:02:21.606 INFO:tasks.workunit.client.0.smithi178.stdout: "num_whiteouts": 0, 2024-09-16T06:02:21.606 INFO:tasks.workunit.client.0.smithi178.stdout: "num_read": 0, 2024-09-16T06:02:21.606 INFO:tasks.workunit.client.0.smithi178.stdout: "num_read_kb": 0, 2024-09-16T06:02:21.606 INFO:tasks.workunit.client.0.smithi178.stdout: "num_write": 2, 2024-09-16T06:02:21.606 INFO:tasks.workunit.client.0.smithi178.stdout: "num_write_kb": 2, 2024-09-16T06:02:21.606 INFO:tasks.workunit.client.0.smithi178.stdout: "num_scrub_errors": 0, 2024-09-16T06:02:21.606 INFO:tasks.workunit.client.0.smithi178.stdout: "num_shallow_scrub_errors": 0, 2024-09-16T06:02:21.606 INFO:tasks.workunit.client.0.smithi178.stdout: "num_deep_scrub_errors": 0, 2024-09-16T06:02:21.606 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_recovered": 0, 2024-09-16T06:02:21.606 INFO:tasks.workunit.client.0.smithi178.stdout: "num_bytes_recovered": 0, 2024-09-16T06:02:21.606 INFO:tasks.workunit.client.0.smithi178.stdout: "num_keys_recovered": 0, 2024-09-16T06:02:21.606 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_omap": 0, 2024-09-16T06:02:21.607 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_hit_set_archive": 0, 2024-09-16T06:02:21.607 INFO:tasks.workunit.client.0.smithi178.stdout: "num_bytes_hit_set_archive": 0, 2024-09-16T06:02:21.607 INFO:tasks.workunit.client.0.smithi178.stdout: "num_flush": 0, 2024-09-16T06:02:21.607 INFO:tasks.workunit.client.0.smithi178.stdout: "num_flush_kb": 0, 2024-09-16T06:02:21.607 INFO:tasks.workunit.client.0.smithi178.stdout: "num_evict": 0, 2024-09-16T06:02:21.607 INFO:tasks.workunit.client.0.smithi178.stdout: "num_evict_kb": 0, 2024-09-16T06:02:21.607 INFO:tasks.workunit.client.0.smithi178.stdout: "num_promote": 0, 2024-09-16T06:02:21.607 INFO:tasks.workunit.client.0.smithi178.stdout: "num_flush_mode_high": 0, 2024-09-16T06:02:21.607 INFO:tasks.workunit.client.0.smithi178.stdout: "num_flush_mode_low": 0, 2024-09-16T06:02:21.607 INFO:tasks.workunit.client.0.smithi178.stdout: "num_evict_mode_some": 0, 2024-09-16T06:02:21.607 INFO:tasks.workunit.client.0.smithi178.stdout: "num_evict_mode_full": 0, 2024-09-16T06:02:21.607 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_pinned": 0, 2024-09-16T06:02:21.607 INFO:tasks.workunit.client.0.smithi178.stdout: "num_legacy_snapsets": 0, 2024-09-16T06:02:21.607 INFO:tasks.workunit.client.0.smithi178.stdout: "num_large_omap_objects": 0, 2024-09-16T06:02:21.608 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_manifest": 0, 2024-09-16T06:02:21.608 INFO:tasks.workunit.client.0.smithi178.stdout: "num_omap_bytes": 0, 2024-09-16T06:02:21.608 INFO:tasks.workunit.client.0.smithi178.stdout: "num_omap_keys": 0, 2024-09-16T06:02:21.608 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_repaired": 0 2024-09-16T06:02:21.608 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.608 INFO:tasks.workunit.client.0.smithi178.stdout: "store_stats": { 2024-09-16T06:02:21.608 INFO:tasks.workunit.client.0.smithi178.stdout: "total": 0, 2024-09-16T06:02:21.608 INFO:tasks.workunit.client.0.smithi178.stdout: "available": 0, 2024-09-16T06:02:21.608 INFO:tasks.workunit.client.0.smithi178.stdout: "internally_reserved": 0, 2024-09-16T06:02:21.608 INFO:tasks.workunit.client.0.smithi178.stdout: "allocated": 0, 2024-09-16T06:02:21.608 INFO:tasks.workunit.client.0.smithi178.stdout: "data_stored": 0, 2024-09-16T06:02:21.608 INFO:tasks.workunit.client.0.smithi178.stdout: "data_compressed": 0, 2024-09-16T06:02:21.608 INFO:tasks.workunit.client.0.smithi178.stdout: "data_compressed_allocated": 0, 2024-09-16T06:02:21.608 INFO:tasks.workunit.client.0.smithi178.stdout: "data_compressed_original": 0, 2024-09-16T06:02:21.609 INFO:tasks.workunit.client.0.smithi178.stdout: "omap_allocated": 0, 2024-09-16T06:02:21.609 INFO:tasks.workunit.client.0.smithi178.stdout: "internal_metadata": 0 2024-09-16T06:02:21.609 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.609 INFO:tasks.workunit.client.0.smithi178.stdout: "log_size": 3, 2024-09-16T06:02:21.609 INFO:tasks.workunit.client.0.smithi178.stdout: "ondisk_log_size": 3, 2024-09-16T06:02:21.609 INFO:tasks.workunit.client.0.smithi178.stdout: "up": 4, 2024-09-16T06:02:21.609 INFO:tasks.workunit.client.0.smithi178.stdout: "acting": 4, 2024-09-16T06:02:21.609 INFO:tasks.workunit.client.0.smithi178.stdout: "num_store_stats": 0 2024-09-16T06:02:21.609 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.609 INFO:tasks.workunit.client.0.smithi178.stdout: "osd_sum": { 2024-09-16T06:02:21.609 INFO:tasks.workunit.client.0.smithi178.stdout: "up_from": 0, 2024-09-16T06:02:21.609 INFO:tasks.workunit.client.0.smithi178.stdout: "seq": 0, 2024-09-16T06:02:21.609 INFO:tasks.workunit.client.0.smithi178.stdout: "num_pgs": 4, 2024-09-16T06:02:21.609 INFO:tasks.workunit.client.0.smithi178.stdout: "num_osds": 1, 2024-09-16T06:02:21.610 INFO:tasks.workunit.client.0.smithi178.stdout: "num_per_pool_osds": 1, 2024-09-16T06:02:21.610 INFO:tasks.workunit.client.0.smithi178.stdout: "num_per_pool_omap_osds": 1, 2024-09-16T06:02:21.610 INFO:tasks.workunit.client.0.smithi178.stdout: "kb": 104857600, 2024-09-16T06:02:21.610 INFO:tasks.workunit.client.0.smithi178.stdout: "kb_used": 26952, 2024-09-16T06:02:21.610 INFO:tasks.workunit.client.0.smithi178.stdout: "kb_used_data": 116, 2024-09-16T06:02:21.610 INFO:tasks.workunit.client.0.smithi178.stdout: "kb_used_omap": 1, 2024-09-16T06:02:21.610 INFO:tasks.workunit.client.0.smithi178.stdout: "kb_used_meta": 26814, 2024-09-16T06:02:21.610 INFO:tasks.workunit.client.0.smithi178.stdout: "kb_avail": 104830648, 2024-09-16T06:02:21.610 INFO:tasks.workunit.client.0.smithi178.stdout: "statfs": { 2024-09-16T06:02:21.610 INFO:tasks.workunit.client.0.smithi178.stdout: "total": 107374182400, 2024-09-16T06:02:21.610 INFO:tasks.workunit.client.0.smithi178.stdout: "available": 107346583552, 2024-09-16T06:02:21.610 INFO:tasks.workunit.client.0.smithi178.stdout: "internally_reserved": 0, 2024-09-16T06:02:21.610 INFO:tasks.workunit.client.0.smithi178.stdout: "allocated": 118784, 2024-09-16T06:02:21.610 INFO:tasks.workunit.client.0.smithi178.stdout: "data_stored": 24786, 2024-09-16T06:02:21.611 INFO:tasks.workunit.client.0.smithi178.stdout: "data_compressed": 0, 2024-09-16T06:02:21.611 INFO:tasks.workunit.client.0.smithi178.stdout: "data_compressed_allocated": 0, 2024-09-16T06:02:21.611 INFO:tasks.workunit.client.0.smithi178.stdout: "data_compressed_original": 0, 2024-09-16T06:02:21.611 INFO:tasks.workunit.client.0.smithi178.stdout: "omap_allocated": 1595, 2024-09-16T06:02:21.611 INFO:tasks.workunit.client.0.smithi178.stdout: "internal_metadata": 27457989 2024-09-16T06:02:21.611 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.611 INFO:tasks.workunit.client.0.smithi178.stdout: "hb_peers": [], 2024-09-16T06:02:21.611 INFO:tasks.workunit.client.0.smithi178.stdout: "snap_trim_queue_len": 0, 2024-09-16T06:02:21.611 INFO:tasks.workunit.client.0.smithi178.stdout: "num_snap_trimming": 0, 2024-09-16T06:02:21.611 INFO:tasks.workunit.client.0.smithi178.stdout: "num_shards_repaired": 0, 2024-09-16T06:02:21.611 INFO:tasks.workunit.client.0.smithi178.stdout: "op_queue_age_hist": { 2024-09-16T06:02:21.611 INFO:tasks.workunit.client.0.smithi178.stdout: "histogram": [], 2024-09-16T06:02:21.611 INFO:tasks.workunit.client.0.smithi178.stdout: "upper_bound": 1 2024-09-16T06:02:21.611 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.612 INFO:tasks.workunit.client.0.smithi178.stdout: "perf_stat": { 2024-09-16T06:02:21.612 INFO:tasks.workunit.client.0.smithi178.stdout: "commit_latency_ms": 0, 2024-09-16T06:02:21.612 INFO:tasks.workunit.client.0.smithi178.stdout: "apply_latency_ms": 0, 2024-09-16T06:02:21.612 INFO:tasks.workunit.client.0.smithi178.stdout: "commit_latency_ns": 0, 2024-09-16T06:02:21.612 INFO:tasks.workunit.client.0.smithi178.stdout: "apply_latency_ns": 0 2024-09-16T06:02:21.612 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.612 INFO:tasks.workunit.client.0.smithi178.stdout: "alerts": [], 2024-09-16T06:02:21.612 INFO:tasks.workunit.client.0.smithi178.stdout: "network_ping_times": [] 2024-09-16T06:02:21.612 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.612 INFO:tasks.workunit.client.0.smithi178.stdout: "osd_sum_by_class": { 2024-09-16T06:02:21.612 INFO:tasks.workunit.client.0.smithi178.stdout: "hdd": { 2024-09-16T06:02:21.612 INFO:tasks.workunit.client.0.smithi178.stdout: "up_from": 0, 2024-09-16T06:02:21.612 INFO:tasks.workunit.client.0.smithi178.stdout: "seq": 0, 2024-09-16T06:02:21.612 INFO:tasks.workunit.client.0.smithi178.stdout: "num_pgs": 4, 2024-09-16T06:02:21.613 INFO:tasks.workunit.client.0.smithi178.stdout: "num_osds": 1, 2024-09-16T06:02:21.613 INFO:tasks.workunit.client.0.smithi178.stdout: "num_per_pool_osds": 1, 2024-09-16T06:02:21.613 INFO:tasks.workunit.client.0.smithi178.stdout: "num_per_pool_omap_osds": 1, 2024-09-16T06:02:21.613 INFO:tasks.workunit.client.0.smithi178.stdout: "kb": 104857600, 2024-09-16T06:02:21.613 INFO:tasks.workunit.client.0.smithi178.stdout: "kb_used": 26952, 2024-09-16T06:02:21.613 INFO:tasks.workunit.client.0.smithi178.stdout: "kb_used_data": 116, 2024-09-16T06:02:21.613 INFO:tasks.workunit.client.0.smithi178.stdout: "kb_used_omap": 1, 2024-09-16T06:02:21.613 INFO:tasks.workunit.client.0.smithi178.stdout: "kb_used_meta": 26814, 2024-09-16T06:02:21.613 INFO:tasks.workunit.client.0.smithi178.stdout: "kb_avail": 104830648, 2024-09-16T06:02:21.613 INFO:tasks.workunit.client.0.smithi178.stdout: "statfs": { 2024-09-16T06:02:21.613 INFO:tasks.workunit.client.0.smithi178.stdout: "total": 107374182400, 2024-09-16T06:02:21.613 INFO:tasks.workunit.client.0.smithi178.stdout: "available": 107346583552, 2024-09-16T06:02:21.613 INFO:tasks.workunit.client.0.smithi178.stdout: "internally_reserved": 0, 2024-09-16T06:02:21.613 INFO:tasks.workunit.client.0.smithi178.stdout: "allocated": 118784, 2024-09-16T06:02:21.613 INFO:tasks.workunit.client.0.smithi178.stdout: "data_stored": 24786, 2024-09-16T06:02:21.614 INFO:tasks.workunit.client.0.smithi178.stdout: "data_compressed": 0, 2024-09-16T06:02:21.614 INFO:tasks.workunit.client.0.smithi178.stdout: "data_compressed_allocated": 0, 2024-09-16T06:02:21.614 INFO:tasks.workunit.client.0.smithi178.stdout: "data_compressed_original": 0, 2024-09-16T06:02:21.614 INFO:tasks.workunit.client.0.smithi178.stdout: "omap_allocated": 1595, 2024-09-16T06:02:21.614 INFO:tasks.workunit.client.0.smithi178.stdout: "internal_metadata": 27457989 2024-09-16T06:02:21.614 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.614 INFO:tasks.workunit.client.0.smithi178.stdout: "hb_peers": [], 2024-09-16T06:02:21.614 INFO:tasks.workunit.client.0.smithi178.stdout: "snap_trim_queue_len": 0, 2024-09-16T06:02:21.614 INFO:tasks.workunit.client.0.smithi178.stdout: "num_snap_trimming": 0, 2024-09-16T06:02:21.614 INFO:tasks.workunit.client.0.smithi178.stdout: "num_shards_repaired": 0, 2024-09-16T06:02:21.614 INFO:tasks.workunit.client.0.smithi178.stdout: "op_queue_age_hist": { 2024-09-16T06:02:21.614 INFO:tasks.workunit.client.0.smithi178.stdout: "histogram": [], 2024-09-16T06:02:21.615 INFO:tasks.workunit.client.0.smithi178.stdout: "upper_bound": 1 2024-09-16T06:02:21.615 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.615 INFO:tasks.workunit.client.0.smithi178.stdout: "perf_stat": { 2024-09-16T06:02:21.615 INFO:tasks.workunit.client.0.smithi178.stdout: "commit_latency_ms": 0, 2024-09-16T06:02:21.615 INFO:tasks.workunit.client.0.smithi178.stdout: "apply_latency_ms": 0, 2024-09-16T06:02:21.615 INFO:tasks.workunit.client.0.smithi178.stdout: "commit_latency_ns": 0, 2024-09-16T06:02:21.615 INFO:tasks.workunit.client.0.smithi178.stdout: "apply_latency_ns": 0 2024-09-16T06:02:21.615 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.615 INFO:tasks.workunit.client.0.smithi178.stdout: "alerts": [], 2024-09-16T06:02:21.615 INFO:tasks.workunit.client.0.smithi178.stdout: "network_ping_times": [] 2024-09-16T06:02:21.615 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:02:21.615 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.615 INFO:tasks.workunit.client.0.smithi178.stdout: "pool_stats": [ 2024-09-16T06:02:21.616 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.616 INFO:tasks.workunit.client.0.smithi178.stdout: "poolid": 1, 2024-09-16T06:02:21.616 INFO:tasks.workunit.client.0.smithi178.stdout: "num_pg": 4, 2024-09-16T06:02:21.616 INFO:tasks.workunit.client.0.smithi178.stdout: "stat_sum": { 2024-09-16T06:02:21.616 INFO:tasks.workunit.client.0.smithi178.stdout: "num_bytes": 19, 2024-09-16T06:02:21.616 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects": 2, 2024-09-16T06:02:21.616 INFO:tasks.workunit.client.0.smithi178.stdout: "num_object_clones": 0, 2024-09-16T06:02:21.616 INFO:tasks.workunit.client.0.smithi178.stdout: "num_object_copies": 4, 2024-09-16T06:02:21.616 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_missing_on_primary": 0, 2024-09-16T06:02:21.616 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_missing": 0, 2024-09-16T06:02:21.616 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_degraded": 2, 2024-09-16T06:02:21.616 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_misplaced": 0, 2024-09-16T06:02:21.616 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_unfound": 0, 2024-09-16T06:02:21.616 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_dirty": 2, 2024-09-16T06:02:21.617 INFO:tasks.workunit.client.0.smithi178.stdout: "num_whiteouts": 0, 2024-09-16T06:02:21.617 INFO:tasks.workunit.client.0.smithi178.stdout: "num_read": 0, 2024-09-16T06:02:21.617 INFO:tasks.workunit.client.0.smithi178.stdout: "num_read_kb": 0, 2024-09-16T06:02:21.617 INFO:tasks.workunit.client.0.smithi178.stdout: "num_write": 2, 2024-09-16T06:02:21.617 INFO:tasks.workunit.client.0.smithi178.stdout: "num_write_kb": 2, 2024-09-16T06:02:21.617 INFO:tasks.workunit.client.0.smithi178.stdout: "num_scrub_errors": 0, 2024-09-16T06:02:21.617 INFO:tasks.workunit.client.0.smithi178.stdout: "num_shallow_scrub_errors": 0, 2024-09-16T06:02:21.617 INFO:tasks.workunit.client.0.smithi178.stdout: "num_deep_scrub_errors": 0, 2024-09-16T06:02:21.617 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_recovered": 0, 2024-09-16T06:02:21.617 INFO:tasks.workunit.client.0.smithi178.stdout: "num_bytes_recovered": 0, 2024-09-16T06:02:21.617 INFO:tasks.workunit.client.0.smithi178.stdout: "num_keys_recovered": 0, 2024-09-16T06:02:21.617 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_omap": 0, 2024-09-16T06:02:21.617 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_hit_set_archive": 0, 2024-09-16T06:02:21.617 INFO:tasks.workunit.client.0.smithi178.stdout: "num_bytes_hit_set_archive": 0, 2024-09-16T06:02:21.618 INFO:tasks.workunit.client.0.smithi178.stdout: "num_flush": 0, 2024-09-16T06:02:21.618 INFO:tasks.workunit.client.0.smithi178.stdout: "num_flush_kb": 0, 2024-09-16T06:02:21.618 INFO:tasks.workunit.client.0.smithi178.stdout: "num_evict": 0, 2024-09-16T06:02:21.618 INFO:tasks.workunit.client.0.smithi178.stdout: "num_evict_kb": 0, 2024-09-16T06:02:21.618 INFO:tasks.workunit.client.0.smithi178.stdout: "num_promote": 0, 2024-09-16T06:02:21.618 INFO:tasks.workunit.client.0.smithi178.stdout: "num_flush_mode_high": 0, 2024-09-16T06:02:21.618 INFO:tasks.workunit.client.0.smithi178.stdout: "num_flush_mode_low": 0, 2024-09-16T06:02:21.618 INFO:tasks.workunit.client.0.smithi178.stdout: "num_evict_mode_some": 0, 2024-09-16T06:02:21.618 INFO:tasks.workunit.client.0.smithi178.stdout: "num_evict_mode_full": 0, 2024-09-16T06:02:21.618 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_pinned": 0, 2024-09-16T06:02:21.618 INFO:tasks.workunit.client.0.smithi178.stdout: "num_legacy_snapsets": 0, 2024-09-16T06:02:21.618 INFO:tasks.workunit.client.0.smithi178.stdout: "num_large_omap_objects": 0, 2024-09-16T06:02:21.618 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_manifest": 0, 2024-09-16T06:02:21.618 INFO:tasks.workunit.client.0.smithi178.stdout: "num_omap_bytes": 0, 2024-09-16T06:02:21.619 INFO:tasks.workunit.client.0.smithi178.stdout: "num_omap_keys": 0, 2024-09-16T06:02:21.619 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_repaired": 0 2024-09-16T06:02:21.619 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.619 INFO:tasks.workunit.client.0.smithi178.stdout: "store_stats": { 2024-09-16T06:02:21.619 INFO:tasks.workunit.client.0.smithi178.stdout: "total": 0, 2024-09-16T06:02:21.619 INFO:tasks.workunit.client.0.smithi178.stdout: "available": 0, 2024-09-16T06:02:21.619 INFO:tasks.workunit.client.0.smithi178.stdout: "internally_reserved": 0, 2024-09-16T06:02:21.619 INFO:tasks.workunit.client.0.smithi178.stdout: "allocated": 4096, 2024-09-16T06:02:21.619 INFO:tasks.workunit.client.0.smithi178.stdout: "data_stored": 19, 2024-09-16T06:02:21.619 INFO:tasks.workunit.client.0.smithi178.stdout: "data_compressed": 0, 2024-09-16T06:02:21.620 INFO:tasks.workunit.client.0.smithi178.stdout: "data_compressed_allocated": 0, 2024-09-16T06:02:21.620 INFO:tasks.workunit.client.0.smithi178.stdout: "data_compressed_original": 0, 2024-09-16T06:02:21.620 INFO:tasks.workunit.client.0.smithi178.stdout: "omap_allocated": 0, 2024-09-16T06:02:21.620 INFO:tasks.workunit.client.0.smithi178.stdout: "internal_metadata": 0 2024-09-16T06:02:21.620 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.620 INFO:tasks.workunit.client.0.smithi178.stdout: "log_size": 3, 2024-09-16T06:02:21.620 INFO:tasks.workunit.client.0.smithi178.stdout: "ondisk_log_size": 3, 2024-09-16T06:02:21.620 INFO:tasks.workunit.client.0.smithi178.stdout: "up": 4, 2024-09-16T06:02:21.620 INFO:tasks.workunit.client.0.smithi178.stdout: "acting": 4, 2024-09-16T06:02:21.620 INFO:tasks.workunit.client.0.smithi178.stdout: "num_store_stats": 1 2024-09-16T06:02:21.620 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:02:21.620 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:02:21.620 INFO:tasks.workunit.client.0.smithi178.stdout: "osd_stats": [ 2024-09-16T06:02:21.620 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.621 INFO:tasks.workunit.client.0.smithi178.stdout: "osd": 0, 2024-09-16T06:02:21.621 INFO:tasks.workunit.client.0.smithi178.stdout: "seq": 21474836482 2024-09-16T06:02:21.621 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:02:21.621 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:02:21.621 INFO:tasks.workunit.client.0.smithi178.stdout: "num_pg_by_state": [ 2024-09-16T06:02:21.621 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.621 INFO:tasks.workunit.client.0.smithi178.stdout: "state": "active+undersized+degraded", 2024-09-16T06:02:21.621 INFO:tasks.workunit.client.0.smithi178.stdout: "num": 2 2024-09-16T06:02:21.621 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.621 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.621 INFO:tasks.workunit.client.0.smithi178.stdout: "state": "active+undersized", 2024-09-16T06:02:21.621 INFO:tasks.workunit.client.0.smithi178.stdout: "num": 2 2024-09-16T06:02:21.621 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:02:21.621 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:02:21.621 INFO:tasks.workunit.client.0.smithi178.stdout: "num_pg_by_osd": [ 2024-09-16T06:02:21.622 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.622 INFO:tasks.workunit.client.0.smithi178.stdout: "osd": 0, 2024-09-16T06:02:21.622 INFO:tasks.workunit.client.0.smithi178.stdout: "num_primary_pg": 4, 2024-09-16T06:02:21.622 INFO:tasks.workunit.client.0.smithi178.stdout: "num_acting_pg": 4, 2024-09-16T06:02:21.622 INFO:tasks.workunit.client.0.smithi178.stdout: "num_up_not_acting_pg": 0 2024-09-16T06:02:21.623 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:02:21.623 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:02:21.623 INFO:tasks.workunit.client.0.smithi178.stdout: "purged_snaps": [ 2024-09-16T06:02:21.624 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:02:21.624 INFO:tasks.workunit.client.0.smithi178.stdout: "pool": 1, 2024-09-16T06:02:21.624 INFO:tasks.workunit.client.0.smithi178.stdout: "purged_snaps": {} 2024-09-16T06:02:21.624 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:02:21.624 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:02:21.624 INFO:tasks.workunit.client.0.smithi178.stdout: "servicemap": { 2024-09-16T06:02:21.624 INFO:tasks.workunit.client.0.smithi178.stdout: "epoch": 1, 2024-09-16T06:02:21.624 INFO:tasks.workunit.client.0.smithi178.stdout: "modified": "2024-09-16T06:01:36.021047+0000", 2024-09-16T06:02:21.624 INFO:tasks.workunit.client.0.smithi178.stdout: "services": {} 2024-09-16T06:02:21.624 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:02:21.624 INFO:tasks.workunit.client.0.smithi178.stdout: "mgrstat_first_committed": 1, 2024-09-16T06:02:21.624 INFO:tasks.workunit.client.0.smithi178.stdout: "mgrstat_last_committed": 6, 2024-09-16T06:02:21.624 INFO:tasks.workunit.client.0.smithi178.stdout: "logm_first_committed": 1, 2024-09-16T06:02:21.624 INFO:tasks.workunit.client.0.smithi178.stdout: "logm_last_committed": 76, 2024-09-16T06:02:21.625 INFO:tasks.workunit.client.0.smithi178.stdout: "paxos": { 2024-09-16T06:02:21.625 INFO:tasks.workunit.client.0.smithi178.stdout: "first_committed": 1, 2024-09-16T06:02:21.625 INFO:tasks.workunit.client.0.smithi178.stdout: "last_committed": 102, 2024-09-16T06:02:21.625 INFO:tasks.workunit.client.0.smithi178.stdout: "last_pn": 0, 2024-09-16T06:02:21.625 INFO:tasks.workunit.client.0.smithi178.stdout: "accepted_pn": 0 2024-09-16T06:02:21.625 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:02:21.625 INFO:tasks.workunit.client.0.smithi178.stdout:} 2024-09-16T06:02:21.629 INFO:tasks.workunit.client.0.smithi178.stderr:report 2888011366 2024-09-16T06:02:21.629 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-16T06:02:21.629 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1685: test_wait_for_clean: run_osd td/ceph-helpers 1 2024-09-16T06:02:21.629 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T06:02:21.629 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T06:02:21.629 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T06:02:21.629 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T06:02:21.629 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-09-16T06:02:21.630 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T06:02:21.630 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:02:21.630 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:02:21.630 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:02:21.630 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-16T06:02:21.630 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-16T06:02:21.630 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T06:02:21.630 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T06:02:21.630 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T06:02:21.630 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T06:02:21.630 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:02:21.630 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:02:21.630 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:02:21.631 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:02:21.631 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:02:21.631 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:02:21.631 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:02:21.631 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:02:21.631 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T06:02:21.631 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T06:02:21.631 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T06:02:21.631 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T06:02:21.631 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:02:21.631 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:02:21.631 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:02:21.631 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:02:21.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T06:02:21.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T06:02:21.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-09-16T06:02:21.632 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T06:02:21.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=53073c9e-37ee-4bb8-b923-a02aecd497c7 2024-09-16T06:02:21.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 53073c9e-37ee-4bb8-b923-a02aecd497c7' 2024-09-16T06:02:21.632 INFO:tasks.workunit.client.0.smithi178.stdout:add osd1 53073c9e-37ee-4bb8-b923-a02aecd497c7 2024-09-16T06:02:21.633 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T06:02:21.633 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDtyedm/BVAIxAArl+/J9K4BPpPxZQfIZ67rg== 2024-09-16T06:02:21.633 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDtyedm/BVAIxAArl+/J9K4BPpPxZQfIZ67rg=="}' 2024-09-16T06:02:21.633 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 53073c9e-37ee-4bb8-b923-a02aecd497c7 -i td/ceph-helpers/1/new.json 2024-09-16T06:02:22.036 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:02:22.047 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-09-16T06:02:22.048 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDtyedm/BVAIxAArl+/J9K4BPpPxZQfIZ67rg== --osd-uuid 53073c9e-37ee-4bb8-b923-a02aecd497c7 2024-09-16T06:02:22.077 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:02:22.076+0000 7f751401e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:02:22.101 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:02:22.100+0000 7f751401e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:02:22.102 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:02:22.101+0000 7f751401e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:02:22.102 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:02:22.102+0000 7f751401e640 -1 bdev(0x562673913c00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2024-09-16T06:02:22.102 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:02:22.102+0000 7f751401e640 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-16T06:02:28.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-16T06:02:28.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T06:02:28.031 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd1 key to auth repository 2024-09-16T06:02:28.032 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T06:02:28.032 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T06:02:28.543 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.1 2024-09-16T06:02:28.544 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T06:02:28.544 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T06:02:28.544 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T06:02:28.544 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T06:02:28.544 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T06:02:28.570 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:02:28.569+0000 7f858c41c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:02:28.602 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:02:28.602+0000 7f858c41c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:02:28.603 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:02:28.603+0000 7f858c41c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:02:28.918 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T06:02:28.918 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:02:28.919 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T06:02:28.919 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:02:28.919 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:02:28.919 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:02:28.919 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:02:28.920 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:02:28.920 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:02:28.920 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:02:29.312 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:02:29.468 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:02:29.468+0000 7f858c41c640 -1 Falling back to public interface 2024-09-16T06:02:30.314 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:02:30.314 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:02:30.315 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:02:30.315 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T06:02:30.315 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:02:30.315 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:02:30.734 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:02:31.736 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:02:31.736 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:02:31.736 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:02:31.737 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T06:02:31.737 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:02:31.737 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:02:32.112 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:02:32.162 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:02:32.162+0000 7f858c41c640 -1 osd.1 0 log_to_monitors true 2024-09-16T06:02:33.113 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:02:33.113 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:02:33.114 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T06:02:33.114 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T06:02:33.114 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:02:33.114 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:02:33.574 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:02:34.576 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:02:34.576 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:02:34.576 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T06:02:34.576 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T06:02:34.577 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:02:34.577 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:02:35.089 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:02:36.091 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:02:36.091 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:02:36.091 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T06:02:36.092 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T06:02:36.092 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:02:36.092 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:02:36.512 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:02:37.513 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:02:37.513 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:02:37.514 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T06:02:37.514 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T06:02:37.515 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:02:37.515 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:02:38.019 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:02:39.020 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:02:39.020 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:02:39.021 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T06:02:39.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T06:02:39.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:02:39.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:02:39.436 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:02:40.437 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:02:40.437 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:02:40.438 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T06:02:40.438 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T06:02:40.438 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:02:40.438 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:02:40.971 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:02:41.974 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:02:41.974 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:02:41.974 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T06:02:41.975 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T06:02:41.975 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:02:41.975 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:02:42.411 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:02:43.412 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:02:43.413 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:02:43.413 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T06:02:43.413 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T06:02:43.413 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:02:43.414 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:02:43.798 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:02:44.799 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:02:44.799 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:02:44.799 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T06:02:44.800 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T06:02:44.800 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:02:44.800 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:02:45.366 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:02:46.368 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:02:46.368 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:02:46.368 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T06:02:46.369 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T06:02:46.369 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:02:46.369 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:02:46.898 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:02:47.899 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:02:47.899 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:02:47.900 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T06:02:47.900 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T06:02:47.900 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:02:47.901 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:02:49.022 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:02:50.024 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:02:50.025 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:02:50.025 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T06:02:50.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T06:02:50.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:02:50.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:02:50.809 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:02:51.811 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:02:51.811 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:02:51.811 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T06:02:51.812 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T06:02:51.812 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:02:51.812 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:02:52.511 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:02:53.513 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:02:53.513 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:02:53.513 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T06:02:53.514 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T06:02:53.514 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:02:53.514 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:02:54.059 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:02:55.061 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:02:55.061 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:02:55.061 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T06:02:55.062 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T06:02:55.062 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:02:55.062 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:02:55.524 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:02:56.525 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:02:56.525 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:02:56.526 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T06:02:56.526 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T06:02:56.526 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:02:56.526 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:02:57.011 INFO:tasks.workunit.client.0.smithi178.stdout:osd.1 up in weight 1 up_from 17 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2446495317,v1:127.0.0.1:6811/2446495317] [v2:127.0.0.1:6812/2446495317,v1:127.0.0.1:6813/2446495317] exists,up 53073c9e-37ee-4bb8-b923-a02aecd497c7 2024-09-16T06:02:57.013 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:02:57.013 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:02:57.013 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:02:57.013 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1686: test_wait_for_clean: wait_for_clean 2024-09-16T06:02:57.013 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T06:02:57.013 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T06:02:57.014 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T06:02:57.014 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T06:02:57.014 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T06:02:57.014 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T06:02:57.014 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T06:02:57.014 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T06:02:57.014 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T06:02:57.098 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T06:02:57.099 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T06:02:57.099 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T06:02:57.099 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T06:02:57.099 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-16T06:02:57.099 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-16T06:02:57.545 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-16T06:02:57.545 INFO:tasks.workunit.client.0.smithi178.stderr:1' 2024-09-16T06:02:57.545 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-16T06:02:57.545 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T06:02:57.545 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T06:02:57.659 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836491 2024-09-16T06:02:57.659 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836491 2024-09-16T06:02:57.659 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836491' 2024-09-16T06:02:57.659 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T06:02:57.659 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T06:02:57.773 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=73014444038 2024-09-16T06:02:57.773 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 73014444038 2024-09-16T06:02:57.773 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836491 1-73014444038' 2024-09-16T06:02:57.774 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T06:02:57.774 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836491 2024-09-16T06:02:57.774 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T06:02:57.774 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-16T06:02:57.775 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836491 2024-09-16T06:02:57.775 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T06:02:57.775 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836491 2024-09-16T06:02:57.776 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.0 seq 21474836491 2024-09-16T06:02:57.776 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2024-09-16T06:02:57.776 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T06:02:58.144 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836489 -lt 21474836491 2024-09-16T06:02:58.144 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-16T06:02:59.145 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T06:02:59.145 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T06:02:59.519 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836491 2024-09-16T06:02:59.520 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T06:02:59.520 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-73014444038 2024-09-16T06:02:59.520 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T06:02:59.521 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-09-16T06:02:59.521 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-73014444038 2024-09-16T06:02:59.521 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T06:02:59.522 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=73014444038 2024-09-16T06:02:59.522 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.1 seq 73014444038 2024-09-16T06:02:59.523 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 73014444038' 2024-09-16T06:02:59.523 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T06:02:59.894 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 73014444038 -lt 73014444038 2024-09-16T06:02:59.895 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T06:02:59.895 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T06:02:59.895 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T06:03:00.393 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T06:03:00.393 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T06:03:00.393 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T06:03:00.393 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T06:03:00.394 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T06:03:00.394 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T06:03:00.394 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T06:03:00.394 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T06:03:00.802 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T06:03:00.803 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T06:03:00.803 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T06:03:00.803 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T06:03:01.310 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T06:03:01.311 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T06:03:01.311 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T06:03:01.311 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: test_wait_for_clean: teardown td/ceph-helpers 2024-09-16T06:03:01.311 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T06:03:01.311 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T06:03:01.311 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T06:03:01.311 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:03:01.311 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:03:01.311 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:03:01.311 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:03:01.311 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:03:01.416 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:03:01.417 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:03:01.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:03:01.417 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:03:01.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:03:01.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:03:01.419 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:03:01.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:03:01.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:03:01.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:03:01.420 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:03:01.421 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:03:01.422 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T06:03:01.422 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T06:03:01.432 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:03:01.432 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:03:01.432 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:03:01.432 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T06:03:01.434 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:03:01.434 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:03:01.435 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T06:03:01.435 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_wait_for_health_ok td/ceph-helpers 2024-09-16T06:03:01.435 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1806: test_wait_for_health_ok: local dir=td/ceph-helpers 2024-09-16T06:03:01.435 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1808: test_wait_for_health_ok: setup td/ceph-helpers 2024-09-16T06:03:01.435 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T06:03:01.435 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T06:03:01.435 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T06:03:01.435 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T06:03:01.435 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T06:03:01.436 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:03:01.436 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:03:01.436 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:03:01.436 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:03:01.436 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:03:01.437 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:03:01.437 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:03:01.438 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:03:01.438 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:03:01.438 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:03:01.438 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:03:01.438 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:03:01.439 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:03:01.439 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:03:01.439 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:03:01.439 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:03:01.439 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:03:01.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T06:03:01.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T06:03:01.441 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:03:01.441 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:03:01.442 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:03:01.442 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T06:03:01.442 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:03:01.442 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:03:01.442 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T06:03:01.444 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T06:03:01.444 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:03:01.444 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:03:01.444 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T06:03:01.446 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T06:03:01.446 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T06:03:01.446 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T06:03:01.446 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T06:03:01.446 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1809: test_wait_for_health_ok: run_mon td/ceph-helpers a --osd_failsafe_full_ratio=.99 --mon_pg_warn_min_per_osd=0 2024-09-16T06:03:01.446 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T06:03:01.446 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T06:03:01.446 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T06:03:01.447 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T06:03:01.447 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T06:03:01.447 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_failsafe_full_ratio=.99 --mon_pg_warn_min_per_osd=0 2024-09-16T06:03:01.822 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T06:03:01.822 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:03:01.822 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:03:01.822 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:03:01.822 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:03:01.823 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:03:01.823 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:03:01.823 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_failsafe_full_ratio=.99 --mon_pg_warn_min_per_osd=0 2024-09-16T06:03:02.073 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T06:03:02.074 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T06:03:02.074 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T06:03:02.074 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T06:03:02.074 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T06:03:02.074 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T06:03:02.074 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T06:03:02.074 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T06:03:02.074 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T06:03:02.075 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:03:02.075 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:03:02.075 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:03:02.075 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T06:03:02.075 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T06:03:02.075 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T06:03:02.153 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T06:03:02.153 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T06:03:02.153 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T06:03:02.153 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T06:03:02.153 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T06:03:02.154 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T06:03:02.154 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T06:03:02.154 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T06:03:02.154 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:03:02.154 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:03:02.154 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:03:02.154 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T06:03:02.154 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T06:03:02.154 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T06:03:02.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1810: test_wait_for_health_ok: run_mgr td/ceph-helpers x --mon_pg_warn_min_per_osd=0 2024-09-16T06:03:02.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T06:03:02.231 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T06:03:02.231 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T06:03:02.231 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T06:03:02.231 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T06:03:02.231 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T06:03:03.068 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T06:03:03.068 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:03:03.068 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:03:03.068 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:03:03.068 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:03:03.068 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:03:03.069 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:03:03.069 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T06:03:03.069 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mon_pg_warn_min_per_osd=0 2024-09-16T06:03:03.099 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1812: test_wait_for_health_ok: run_osd td/ceph-helpers 0 2024-09-16T06:03:03.100 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T06:03:03.100 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T06:03:03.100 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T06:03:03.100 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T06:03:03.100 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T06:03:03.100 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T06:03:03.100 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:03:03.100 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:03:03.100 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:03:03.101 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T06:03:03.101 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T06:03:03.101 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T06:03:03.101 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T06:03:03.101 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T06:03:03.101 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T06:03:03.101 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:03:03.101 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:03:03.101 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:03:03.101 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:03:03.102 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:03:03.102 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:03:03.102 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:03:03.102 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:03:03.102 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T06:03:03.102 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T06:03:03.102 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T06:03:03.102 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T06:03:03.102 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:03:03.102 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:03:03.102 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:03:03.102 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:03:03.102 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T06:03:03.103 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T06:03:03.103 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T06:03:03.103 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T06:03:03.103 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bf964e75-a40c-4e9b-9c91-3b255ef32646 2024-09-16T06:03:03.103 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 bf964e75-a40c-4e9b-9c91-3b255ef32646' 2024-09-16T06:03:03.104 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 bf964e75-a40c-4e9b-9c91-3b255ef32646 2024-09-16T06:03:03.104 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T06:03:03.120 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAXyudmH5AVBxAABcOYEa1nZDq+FJ4YIulIbg== 2024-09-16T06:03:03.120 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAXyudmH5AVBxAABcOYEa1nZDq+FJ4YIulIbg=="}' 2024-09-16T06:03:03.120 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bf964e75-a40c-4e9b-9c91-3b255ef32646 -i td/ceph-helpers/0/new.json 2024-09-16T06:03:03.380 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:03:03.389 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T06:03:03.392 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAXyudmH5AVBxAABcOYEa1nZDq+FJ4YIulIbg== --osd-uuid bf964e75-a40c-4e9b-9c91-3b255ef32646 2024-09-16T06:03:03.420 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:03:03.419+0000 7fee97e60640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:03:03.449 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:03:03.448+0000 7fee97e60640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:03:03.450 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:03:03.449+0000 7fee97e60640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:03:03.450 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:03:03.450+0000 7fee97e60640 -1 bdev(0x55fd28419c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-16T06:03:03.450 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:03:03.450+0000 7fee97e60640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T06:03:09.196 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T06:03:09.196 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T06:03:09.197 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T06:03:09.197 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T06:03:09.197 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T06:03:09.712 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T06:03:09.713 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T06:03:09.713 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T06:03:09.714 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T06:03:09.714 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T06:03:09.714 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T06:03:09.739 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:03:09.739+0000 7f356ba1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:03:09.760 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:03:09.759+0000 7f356ba1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:03:09.761 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:03:09.760+0000 7f356ba1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:03:10.165 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T06:03:10.165 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:03:10.165 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T06:03:10.165 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:03:10.165 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:03:10.165 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:03:10.166 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:03:10.166 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:03:10.166 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:03:10.166 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:03:10.537 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:03:10.619 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:03:10.619+0000 7f356ba1e640 -1 Falling back to public interface 2024-09-16T06:03:11.538 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:03:11.538 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:03:11.538 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T06:03:11.539 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:03:11.539 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:03:11.539 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:03:11.912 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:03:12.913 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:03:12.913 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:03:12.913 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:03:12.914 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T06:03:12.914 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:03:12.914 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:03:13.297 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:03:13.695 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:03:13.694+0000 7f356ba1e640 -1 osd.0 0 log_to_monitors true 2024-09-16T06:03:14.299 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:03:14.299 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:03:14.299 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T06:03:14.300 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T06:03:14.300 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:03:14.300 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:03:14.769 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:03:15.770 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:03:15.770 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:03:15.771 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T06:03:15.771 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T06:03:15.771 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:03:15.771 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:03:16.177 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:03:17.179 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:03:17.179 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:03:17.179 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T06:03:17.179 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T06:03:17.180 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:03:17.180 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:03:17.694 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:03:18.695 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:03:18.695 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:03:18.695 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T06:03:18.696 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T06:03:18.696 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:03:18.696 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:03:19.103 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:03:20.104 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:03:20.104 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:03:20.105 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T06:03:20.105 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T06:03:20.105 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:03:20.105 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:03:20.627 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:03:21.629 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:03:21.629 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:03:21.629 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T06:03:21.630 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T06:03:21.630 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:03:21.630 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:03:22.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:03:23.037 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:03:23.037 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:03:23.037 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T06:03:23.038 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T06:03:23.038 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:03:23.038 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:03:23.463 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:03:24.464 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:03:24.465 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:03:24.465 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T06:03:24.465 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T06:03:24.465 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:03:24.466 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:03:24.853 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:03:25.855 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:03:25.855 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:03:25.855 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T06:03:25.856 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T06:03:25.856 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:03:25.856 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:03:26.449 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:03:27.450 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:03:27.450 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:03:27.451 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T06:03:27.451 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T06:03:27.451 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:03:27.451 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:03:28.261 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:03:29.262 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:03:29.263 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:03:29.263 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T06:03:29.263 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T06:03:29.263 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:03:29.263 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:03:30.155 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:03:31.156 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:03:31.156 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:03:31.157 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T06:03:31.157 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T06:03:31.157 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:03:31.157 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:03:31.598 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:03:32.599 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:03:32.600 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:03:32.600 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T06:03:32.601 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T06:03:32.601 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:03:32.601 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:03:33.277 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:03:34.278 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:03:34.278 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:03:34.279 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T06:03:34.279 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T06:03:34.279 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:03:34.279 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:03:35.012 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:03:36.013 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:03:36.014 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:03:36.014 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T06:03:36.014 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T06:03:36.015 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:03:36.015 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:03:36.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:03:37.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:03:37.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:03:37.420 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T06:03:37.421 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T06:03:37.421 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:03:37.421 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:03:37.829 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1364652753,v1:127.0.0.1:6803/1364652753] [v2:127.0.0.1:6804/1364652753,v1:127.0.0.1:6805/1364652753] exists,up bf964e75-a40c-4e9b-9c91-3b255ef32646 2024-09-16T06:03:37.833 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:03:37.834 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:03:37.834 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:03:37.834 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1813: test_wait_for_health_ok: run_osd td/ceph-helpers 1 2024-09-16T06:03:37.834 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T06:03:37.834 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T06:03:37.834 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T06:03:37.834 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T06:03:37.834 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-09-16T06:03:37.834 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T06:03:37.834 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:03:37.834 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:03:37.835 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:03:37.835 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-16T06:03:37.835 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-16T06:03:37.835 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T06:03:37.835 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T06:03:37.835 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T06:03:37.835 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T06:03:37.835 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:03:37.835 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:03:37.835 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:03:37.835 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:03:37.835 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:03:37.836 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:03:37.836 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:03:37.836 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:03:37.836 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T06:03:37.836 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T06:03:37.836 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T06:03:37.836 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T06:03:37.836 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:03:37.836 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:03:37.836 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:03:37.836 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:03:37.837 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T06:03:37.837 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T06:03:37.837 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-09-16T06:03:37.837 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T06:03:37.837 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=79e932da-cd01-4a84-a1e7-79577dafc1a3 2024-09-16T06:03:37.837 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 79e932da-cd01-4a84-a1e7-79577dafc1a3' 2024-09-16T06:03:37.837 INFO:tasks.workunit.client.0.smithi178.stdout:add osd1 79e932da-cd01-4a84-a1e7-79577dafc1a3 2024-09-16T06:03:37.837 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T06:03:37.852 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA5yudmlZG9MhAArdAXEBymjjAHQfG+wPTefA== 2024-09-16T06:03:37.852 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA5yudmlZG9MhAArdAXEBymjjAHQfG+wPTefA=="}' 2024-09-16T06:03:37.852 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 79e932da-cd01-4a84-a1e7-79577dafc1a3 -i td/ceph-helpers/1/new.json 2024-09-16T06:03:38.340 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:03:38.351 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-09-16T06:03:38.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA5yudmlZG9MhAArdAXEBymjjAHQfG+wPTefA== --osd-uuid 79e932da-cd01-4a84-a1e7-79577dafc1a3 2024-09-16T06:03:38.381 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:03:38.380+0000 7f82a7e40640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:03:38.395 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:03:38.395+0000 7f82a7e40640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:03:38.397 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:03:38.396+0000 7f82a7e40640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:03:38.397 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:03:38.396+0000 7f82a7e40640 -1 bdev(0x5589de083c00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2024-09-16T06:03:38.397 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:03:38.397+0000 7f82a7e40640 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-16T06:03:44.150 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-16T06:03:44.151 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T06:03:44.151 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd1 key to auth repository 2024-09-16T06:03:44.151 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T06:03:44.152 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T06:03:44.667 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.1 2024-09-16T06:03:44.668 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T06:03:44.668 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T06:03:44.668 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T06:03:44.669 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T06:03:44.669 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T06:03:44.695 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:03:44.694+0000 7f781e412640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:03:44.717 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:03:44.717+0000 7f781e412640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:03:44.718 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:03:44.718+0000 7f781e412640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:03:45.122 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T06:03:45.122 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:03:45.122 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T06:03:45.122 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:03:45.122 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:03:45.122 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:03:45.122 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:03:45.123 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:03:45.123 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:03:45.123 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:03:45.491 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:03:46.318 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:03:46.318+0000 7f781e412640 -1 Falling back to public interface 2024-09-16T06:03:46.492 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:03:46.492 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:03:46.493 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T06:03:46.493 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:03:46.493 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:03:46.493 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:03:46.907 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:03:47.908 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:03:47.909 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:03:47.909 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:03:47.909 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T06:03:47.910 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:03:47.910 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:03:48.335 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:03:48.996 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:03:48.996+0000 7f781e412640 -1 osd.1 0 log_to_monitors true 2024-09-16T06:03:49.336 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:03:49.336 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:03:49.337 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T06:03:49.337 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T06:03:49.337 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:03:49.337 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:03:49.841 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:03:50.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:03:50.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:03:50.843 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T06:03:50.844 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T06:03:50.844 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:03:50.844 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:03:51.263 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:03:52.264 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:03:52.265 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T06:03:52.265 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:03:52.265 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T06:03:52.265 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:03:52.265 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:03:52.649 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:03:53.650 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:03:53.650 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:03:53.650 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T06:03:53.651 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T06:03:53.651 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:03:53.651 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:03:54.032 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:03:55.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:03:55.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:03:55.034 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T06:03:55.035 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T06:03:55.035 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:03:55.035 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:03:55.469 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:03:56.471 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:03:56.471 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:03:56.471 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T06:03:56.472 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T06:03:56.472 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:03:56.472 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:03:56.913 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:03:57.914 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:03:57.914 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:03:57.914 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T06:03:57.915 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T06:03:57.915 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:03:57.915 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:03:58.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:03:59.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:03:59.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:03:59.443 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T06:03:59.444 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T06:03:59.444 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:03:59.444 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:03:59.978 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:04:00.980 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:04:00.980 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:04:00.980 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T06:04:00.981 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T06:04:00.981 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:04:00.981 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:04:01.491 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:04:02.493 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:04:02.493 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:04:02.493 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T06:04:02.493 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T06:04:02.494 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:04:02.494 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:04:03.415 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:04:04.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:04:04.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:04:04.417 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T06:04:04.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T06:04:04.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:04:04.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:04:05.264 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:04:06.263 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:04:06.263 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:04:06.263 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T06:04:06.263 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T06:04:06.264 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:04:06.264 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:04:06.957 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:04:07.959 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:04:07.959 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:04:07.959 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T06:04:07.959 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T06:04:07.960 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:04:07.960 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:04:08.740 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:04:09.742 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:04:09.742 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:04:09.743 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T06:04:09.744 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T06:04:09.744 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:04:09.744 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:04:10.383 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:04:11.384 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:04:11.385 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:04:11.385 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T06:04:11.385 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T06:04:11.385 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:04:11.386 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:04:12.148 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:04:13.149 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:04:13.149 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:04:13.150 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T06:04:13.150 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T06:04:13.150 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:04:13.150 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:04:13.532 INFO:tasks.workunit.client.0.smithi178.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2512617331,v1:127.0.0.1:6811/2512617331] [v2:127.0.0.1:6812/2512617331,v1:127.0.0.1:6813/2512617331] exists,up 79e932da-cd01-4a84-a1e7-79577dafc1a3 2024-09-16T06:04:13.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:04:13.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:04:13.537 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:04:13.537 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1814: test_wait_for_health_ok: run_osd td/ceph-helpers 2 2024-09-16T06:04:13.537 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T06:04:13.537 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T06:04:13.537 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T06:04:13.537 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T06:04:13.537 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/2 2024-09-16T06:04:13.537 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T06:04:13.537 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:04:13.538 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:04:13.538 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:04:13.538 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2024-09-16T06:04:13.538 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2024-09-16T06:04:13.538 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T06:04:13.538 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T06:04:13.538 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T06:04:13.538 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T06:04:13.538 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:04:13.538 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:04:13.539 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:04:13.539 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:04:13.539 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:04:13.539 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:04:13.539 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:04:13.539 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:04:13.539 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T06:04:13.539 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T06:04:13.540 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T06:04:13.540 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T06:04:13.540 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:04:13.540 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:04:13.540 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:04:13.540 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:04:13.540 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T06:04:13.540 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T06:04:13.540 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/2 2024-09-16T06:04:13.540 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T06:04:13.541 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e901a448-867a-47bc-86b4-86cdc896aef0 2024-09-16T06:04:13.541 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 e901a448-867a-47bc-86b4-86cdc896aef0' 2024-09-16T06:04:13.541 INFO:tasks.workunit.client.0.smithi178.stdout:add osd2 e901a448-867a-47bc-86b4-86cdc896aef0 2024-09-16T06:04:13.541 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T06:04:13.554 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBdyudm7CkCIRAAH14GcQ/vd1itW+ZiJMz0dQ== 2024-09-16T06:04:13.554 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBdyudm7CkCIRAAH14GcQ/vd1itW+ZiJMz0dQ=="}' 2024-09-16T06:04:13.554 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e901a448-867a-47bc-86b4-86cdc896aef0 -i td/ceph-helpers/2/new.json 2024-09-16T06:04:14.142 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:04:14.154 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/2/new.json 2024-09-16T06:04:14.155 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBdyudm7CkCIRAAH14GcQ/vd1itW+ZiJMz0dQ== --osd-uuid e901a448-867a-47bc-86b4-86cdc896aef0 2024-09-16T06:04:14.185 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:04:14.184+0000 7f700d80c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:04:14.201 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:04:14.200+0000 7f700d80c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:04:14.202 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:04:14.202+0000 7f700d80c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:04:14.202 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:04:14.202+0000 7f700d80c640 -1 bdev(0x55d358627c00 td/ceph-helpers/2/block) open stat got: (1) Operation not permitted 2024-09-16T06:04:14.202 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:04:14.202+0000 7f700d80c640 -1 bluestore(td/ceph-helpers/2) _read_fsid unparsable uuid 2024-09-16T06:04:20.913 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/2/keyring 2024-09-16T06:04:20.913 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T06:04:20.914 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd2 key to auth repository 2024-09-16T06:04:20.914 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T06:04:20.915 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T06:04:21.435 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.2 2024-09-16T06:04:21.435 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T06:04:21.436 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T06:04:21.436 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T06:04:21.436 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T06:04:21.436 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T06:04:21.462 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:04:21.461+0000 7f1148815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:04:21.477 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:04:21.476+0000 7f1148815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:04:21.477 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:04:21.477+0000 7f1148815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:04:21.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T06:04:21.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:04:21.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T06:04:21.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:04:21.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:04:21.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:04:21.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:04:21.883 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:04:21.883 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:04:21.883 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:04:22.299 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:04:22.574 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:04:22.574+0000 7f1148815640 -1 Falling back to public interface 2024-09-16T06:04:23.300 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:04:23.300 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:04:23.301 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T06:04:23.301 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:04:23.301 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:04:23.301 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:04:23.681 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:04:24.682 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:04:24.682 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:04:24.683 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:04:24.683 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T06:04:24.683 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:04:24.683 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:04:25.068 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:04:25.731 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:04:25.729+0000 7f1148815640 -1 osd.2 0 log_to_monitors true 2024-09-16T06:04:26.069 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:04:26.069 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:04:26.070 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T06:04:26.070 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T06:04:26.070 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:04:26.070 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:04:26.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:04:27.554 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:04:27.554 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:04:27.555 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T06:04:27.555 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T06:04:27.555 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:04:27.555 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:04:27.940 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:04:28.941 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:04:28.941 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:04:28.942 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T06:04:28.942 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T06:04:28.942 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:04:28.942 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:04:29.432 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:04:30.433 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:04:30.434 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T06:04:30.434 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:04:30.434 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T06:04:30.434 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:04:30.434 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:04:30.904 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:04:31.905 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:04:31.905 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:04:31.905 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T06:04:31.906 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T06:04:31.906 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:04:31.906 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:04:32.285 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:04:33.287 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:04:33.287 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:04:33.287 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T06:04:33.288 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T06:04:33.288 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:04:33.288 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:04:33.683 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:04:34.684 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:04:34.685 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:04:34.685 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T06:04:34.685 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T06:04:34.685 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:04:34.685 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:04:35.063 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:04:36.064 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:04:36.064 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:04:36.065 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T06:04:36.065 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T06:04:36.065 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:04:36.065 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:04:36.557 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:04:37.558 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:04:37.558 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:04:37.558 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T06:04:37.559 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T06:04:37.559 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:04:37.559 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:04:38.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:04:39.033 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:04:39.033 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:04:39.034 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T06:04:39.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T06:04:39.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:04:39.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:04:39.498 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:04:40.499 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:04:40.500 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:04:40.500 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T06:04:40.500 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T06:04:40.500 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:04:40.501 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:04:41.125 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:04:42.126 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:04:42.126 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:04:42.127 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T06:04:42.127 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T06:04:42.127 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:04:42.127 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:04:42.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:04:43.796 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:04:43.796 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:04:43.796 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T06:04:43.797 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T06:04:43.797 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:04:43.797 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:04:44.943 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:04:45.944 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:04:45.945 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:04:45.945 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T06:04:45.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T06:04:45.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:04:45.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:04:46.695 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:04:47.696 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:04:47.697 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:04:47.697 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T06:04:47.697 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T06:04:47.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:04:47.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:04:48.455 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:04:49.457 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:04:49.457 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:04:49.457 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T06:04:49.458 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T06:04:49.458 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:04:49.458 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:04:49.836 INFO:tasks.workunit.client.0.smithi178.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1234626848,v1:127.0.0.1:6819/1234626848] [v2:127.0.0.1:6820/1234626848,v1:127.0.0.1:6821/1234626848] exists,up e901a448-867a-47bc-86b4-86cdc896aef0 2024-09-16T06:04:49.837 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:04:49.837 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:04:49.837 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:04:49.837 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1815: test_wait_for_health_ok: kill_daemons td/ceph-helpers TERM osd 2024-09-16T06:04:49.838 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:04:49.838 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:04:49.838 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:04:49.838 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:04:49.838 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:04:51.145 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:04:51.145 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1816: test_wait_for_health_ok: ceph osd down 0 2024-09-16T06:04:51.573 INFO:tasks.workunit.client.0.smithi178.stderr:osd.0 is already down. 2024-09-16T06:04:51.585 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1818: test_wait_for_health_ok: TIMEOUT=1 2024-09-16T06:04:51.585 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1818: test_wait_for_health_ok: wait_for_health_ok 2024-09-16T06:04:51.585 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1802: wait_for_health_ok: wait_for_health HEALTH_OK 2024-09-16T06:04:51.585 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1781: wait_for_health: local grepstr=HEALTH_OK 2024-09-16T06:04:51.585 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1782: wait_for_health: get_timeout_delays 1 .1 2024-09-16T06:04:51.586 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T06:04:51.586 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T06:04:51.586 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T06:04:51.586 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T06:04:51.586 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T06:04:51.612 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1782: wait_for_health: delays=('0.1' '0.2' '0.4' '0.3') 2024-09-16T06:04:51.612 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1782: wait_for_health: local -a delays 2024-09-16T06:04:51.612 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1783: wait_for_health: local -i loop=0 2024-09-16T06:04:51.612 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: wait_for_health: ceph health detail 2024-09-16T06:04:51.612 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: wait_for_health: grep HEALTH_OK 2024-09-16T06:04:52.145 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1786: wait_for_health: (( 0 >= 4 )) 2024-09-16T06:04:52.146 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_health: sleep 0.1 2024-09-16T06:04:52.247 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_health: loop+=1 2024-09-16T06:04:52.247 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: wait_for_health: ceph health detail 2024-09-16T06:04:52.247 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: wait_for_health: grep HEALTH_OK 2024-09-16T06:04:52.744 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1786: wait_for_health: (( 1 >= 4 )) 2024-09-16T06:04:52.745 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_health: sleep 0.2 2024-09-16T06:04:52.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_health: loop+=1 2024-09-16T06:04:52.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: wait_for_health: ceph health detail 2024-09-16T06:04:52.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: wait_for_health: grep HEALTH_OK 2024-09-16T06:04:53.450 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1786: wait_for_health: (( 2 >= 4 )) 2024-09-16T06:04:53.450 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_health: sleep 0.4 2024-09-16T06:04:53.851 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_health: loop+=1 2024-09-16T06:04:53.852 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: wait_for_health: ceph health detail 2024-09-16T06:04:53.852 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: wait_for_health: grep HEALTH_OK 2024-09-16T06:04:54.351 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1786: wait_for_health: (( 3 >= 4 )) 2024-09-16T06:04:54.351 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_health: sleep 0.3 2024-09-16T06:04:54.652 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_health: loop+=1 2024-09-16T06:04:54.653 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: wait_for_health: ceph health detail 2024-09-16T06:04:54.653 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: wait_for_health: grep HEALTH_OK 2024-09-16T06:04:55.153 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1786: wait_for_health: (( 4 >= 4 )) 2024-09-16T06:04:55.154 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1787: wait_for_health: ceph health detail 2024-09-16T06:04:55.646 INFO:tasks.workunit.client.0.smithi178.stdout:HEALTH_WARN 3 osds down; 1 host (3 osds) down; 1 root (3 osds) down 2024-09-16T06:04:55.646 INFO:tasks.workunit.client.0.smithi178.stdout:[WRN] OSD_DOWN: 3 osds down 2024-09-16T06:04:55.646 INFO:tasks.workunit.client.0.smithi178.stdout: osd.0 (root=default,host=smithi178) is down 2024-09-16T06:04:55.646 INFO:tasks.workunit.client.0.smithi178.stdout: osd.1 (root=default,host=smithi178) is down 2024-09-16T06:04:55.646 INFO:tasks.workunit.client.0.smithi178.stdout: osd.2 (root=default,host=smithi178) is down 2024-09-16T06:04:55.646 INFO:tasks.workunit.client.0.smithi178.stdout:[WRN] OSD_HOST_DOWN: 1 host (3 osds) down 2024-09-16T06:04:55.646 INFO:tasks.workunit.client.0.smithi178.stdout: host smithi178 (root=default) (3 osds) is down 2024-09-16T06:04:55.646 INFO:tasks.workunit.client.0.smithi178.stdout:[WRN] OSD_ROOT_DOWN: 1 root (3 osds) down 2024-09-16T06:04:55.647 INFO:tasks.workunit.client.0.smithi178.stdout: root default (3 osds) is down 2024-09-16T06:04:55.660 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1788: wait_for_health: return 1 2024-09-16T06:04:55.660 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1802: wait_for_health_ok: return 1 2024-09-16T06:04:55.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1820: test_wait_for_health_ok: activate_osd td/ceph-helpers 0 2024-09-16T06:04:55.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-09-16T06:04:55.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T06:04:55.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T06:04:55.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T06:04:55.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-09-16T06:04:55.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T06:04:55.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:04:55.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:04:55.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:04:55.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T06:04:55.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T06:04:55.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T06:04:55.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T06:04:55.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T06:04:55.662 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T06:04:55.662 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:04:55.662 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:04:55.662 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:04:55.662 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:04:55.662 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:04:55.662 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:04:55.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:04:55.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:04:55.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T06:04:55.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T06:04:55.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:04:55.663 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:04:55.663 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:04:55.663 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:04:55.663 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T06:04:55.663 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T06:04:55.663 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-09-16T06:04:55.663 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T06:04:55.663 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T06:04:55.664 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T06:04:55.664 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-09-16T06:04:55.664 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T06:04:55.664 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T06:04:55.664 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T06:04:55.664 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T06:04:55.686 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:04:55.685+0000 7f733f5d7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:04:55.706 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:04:55.705+0000 7f733f5d7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:04:55.707 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:04:55.706+0000 7f733f5d7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:04:56.136 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T06:04:56.137 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:04:56.137 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T06:04:56.137 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:04:56.137 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:04:56.137 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:04:56.137 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:04:56.137 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:04:56.138 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:04:56.138 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:04:56.506 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:04:57.507 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:04:57.507 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:04:57.508 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:04:57.508 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T06:04:57.508 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:04:57.508 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:04:57.576 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:04:57.576+0000 7f733f5d7640 -1 Falling back to public interface 2024-09-16T06:04:57.922 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:04:58.923 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:04:58.924 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:04:58.924 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:04:58.924 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T06:04:58.925 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:04:58.925 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:04:59.303 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:05:00.305 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:05:00.305 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:05:00.305 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T06:05:00.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T06:05:00.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:05:00.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:05:00.492 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:05:00.492+0000 7f733f5d7640 -1 osd.0 16 log_to_monitors true 2024-09-16T06:05:00.752 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:05:01.753 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:05:01.753 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:05:01.753 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T06:05:01.754 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T06:05:01.754 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:05:01.754 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:05:02.123 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 19 up_thru 0 down_at 17 last_clean_interval [5,16) [v2:127.0.0.1:6802/2665342358,v1:127.0.0.1:6803/2665342358] [v2:127.0.0.1:6804/2665342358,v1:127.0.0.1:6805/2665342358] exists,up bf964e75-a40c-4e9b-9c91-3b255ef32646 2024-09-16T06:05:02.127 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:05:02.127 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:05:02.127 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:05:02.127 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1821: test_wait_for_health_ok: activate_osd td/ceph-helpers 1 2024-09-16T06:05:02.127 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-09-16T06:05:02.127 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T06:05:02.127 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-16T06:05:02.127 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T06:05:02.127 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/1 2024-09-16T06:05:02.127 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T06:05:02.128 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:05:02.128 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:05:02.128 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:05:02.128 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-16T06:05:02.128 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-16T06:05:02.128 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T06:05:02.128 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T06:05:02.128 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T06:05:02.128 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T06:05:02.128 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:05:02.128 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:05:02.128 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:05:02.128 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:05:02.129 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:05:02.129 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:05:02.129 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:05:02.129 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:05:02.129 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T06:05:02.129 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T06:05:02.129 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:05:02.129 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:05:02.129 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:05:02.130 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:05:02.130 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T06:05:02.130 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T06:05:02.130 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/1 2024-09-16T06:05:02.130 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-16T06:05:02.130 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.1 2024-09-16T06:05:02.131 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T06:05:02.131 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/1/whoami 2024-09-16T06:05:02.131 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-16T06:05:02.131 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T06:05:02.132 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T06:05:02.132 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T06:05:02.152 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:05:02.151+0000 7fb3db211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:05:02.154 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:05:02.154+0000 7fb3db211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:05:02.156 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:05:02.155+0000 7fb3db211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:05:02.506 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-16T06:05:02.506 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:05:02.506 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T06:05:02.506 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:05:02.506 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:05:02.506 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:05:02.506 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:05:02.507 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:05:02.507 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:05:02.507 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:05:02.875 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:05:03.485 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:05:03.485+0000 7fb3db211640 -1 Falling back to public interface 2024-09-16T06:05:03.877 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:05:03.877 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:05:03.877 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T06:05:03.878 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:05:03.878 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:05:03.878 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:05:04.293 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:05:05.295 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:05:05.295 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:05:05.295 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:05:05.296 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T06:05:05.296 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:05:05.296 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:05:05.737 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:05:06.547 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:05:06.547+0000 7fb3db211640 -1 osd.1 14 log_to_monitors true 2024-09-16T06:05:06.738 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:05:06.738 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:05:06.738 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T06:05:06.739 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T06:05:06.739 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:05:06.739 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:05:07.154 INFO:tasks.workunit.client.0.smithi178.stdout:osd.1 up in weight 1 up_from 21 up_thru 0 down_at 16 last_clean_interval [10,14) [v2:127.0.0.1:6810/1973187393,v1:127.0.0.1:6811/1973187393] [v2:127.0.0.1:6812/1973187393,v1:127.0.0.1:6813/1973187393] exists,up 79e932da-cd01-4a84-a1e7-79577dafc1a3 2024-09-16T06:05:07.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:05:07.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:05:07.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:05:07.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1822: test_wait_for_health_ok: activate_osd td/ceph-helpers 2 2024-09-16T06:05:07.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-09-16T06:05:07.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T06:05:07.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-16T06:05:07.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T06:05:07.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/2 2024-09-16T06:05:07.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T06:05:07.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:05:07.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:05:07.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:05:07.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2024-09-16T06:05:07.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2024-09-16T06:05:07.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T06:05:07.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T06:05:07.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T06:05:07.159 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T06:05:07.159 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:05:07.159 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:05:07.160 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:05:07.160 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:05:07.160 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:05:07.160 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:05:07.160 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:05:07.160 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:05:07.160 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T06:05:07.160 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T06:05:07.160 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:05:07.160 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:05:07.160 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:05:07.160 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:05:07.160 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T06:05:07.161 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T06:05:07.161 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/2 2024-09-16T06:05:07.161 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-16T06:05:07.161 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.2 2024-09-16T06:05:07.162 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T06:05:07.162 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/2/whoami 2024-09-16T06:05:07.162 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-16T06:05:07.162 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T06:05:07.162 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T06:05:07.162 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T06:05:07.184 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:05:07.184+0000 7fea6cb41640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:05:07.187 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:05:07.186+0000 7fea6cb41640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:05:07.188 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:05:07.187+0000 7fea6cb41640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:05:07.542 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-16T06:05:07.542 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:05:07.542 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T06:05:07.542 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:05:07.542 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:05:07.542 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:05:07.542 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:05:07.542 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:05:07.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:05:07.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:05:07.951 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:05:08.537 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:05:08.537+0000 7fea6cb41640 -1 Falling back to public interface 2024-09-16T06:05:08.953 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:05:08.953 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:05:08.953 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T06:05:08.953 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:05:08.954 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:05:08.954 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:05:09.348 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:05:10.350 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:05:10.350 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:05:10.350 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T06:05:10.350 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:05:10.350 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:05:10.351 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:05:10.778 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:05:11.779 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:05:11.780 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:05:11.780 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T06:05:11.780 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T06:05:11.780 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:05:11.780 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:05:11.788 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:05:11.787+0000 7fea6cb41640 -1 osd.2 16 log_to_monitors true 2024-09-16T06:05:12.190 INFO:tasks.workunit.client.0.smithi178.stdout:osd.2 up in weight 1 up_from 23 up_thru 0 down_at 17 last_clean_interval [15,16) [v2:127.0.0.1:6818/1604288040,v1:127.0.0.1:6819/1604288040] [v2:127.0.0.1:6820/1604288040,v1:127.0.0.1:6821/1604288040] exists,up e901a448-867a-47bc-86b4-86cdc896aef0 2024-09-16T06:05:12.191 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:05:12.191 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:05:12.192 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:05:12.192 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1823: test_wait_for_health_ok: wait_for_health_ok 2024-09-16T06:05:12.192 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1802: wait_for_health_ok: wait_for_health HEALTH_OK 2024-09-16T06:05:12.192 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1781: wait_for_health: local grepstr=HEALTH_OK 2024-09-16T06:05:12.192 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1782: wait_for_health: get_timeout_delays 300 .1 2024-09-16T06:05:12.192 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T06:05:12.192 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T06:05:12.192 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T06:05:12.192 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T06:05:12.192 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T06:05:12.369 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1782: wait_for_health: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '4.5') 2024-09-16T06:05:12.370 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1782: wait_for_health: local -a delays 2024-09-16T06:05:12.370 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1783: wait_for_health: local -i loop=0 2024-09-16T06:05:12.370 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: wait_for_health: ceph health detail 2024-09-16T06:05:12.370 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: wait_for_health: grep HEALTH_OK 2024-09-16T06:05:12.872 INFO:tasks.workunit.client.0.smithi178.stdout:HEALTH_OK 2024-09-16T06:05:12.873 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1824: test_wait_for_health_ok: teardown td/ceph-helpers 2024-09-16T06:05:12.874 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T06:05:12.874 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T06:05:12.874 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T06:05:12.874 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:05:12.874 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:05:12.874 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:05:12.874 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:05:12.874 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:05:12.981 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:05:12.981 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:05:12.982 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:05:12.982 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:05:12.983 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:05:12.983 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:05:12.983 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:05:12.984 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:05:12.984 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:05:12.985 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:05:12.985 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:05:12.985 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:05:12.986 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T06:05:12.986 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T06:05:13.002 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:05:13.002 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:05:13.002 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:05:13.002 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T06:05:13.002 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:05:13.003 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:05:13.003 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T06:05:13.003 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_wait_for_osd td/ceph-helpers 2024-09-16T06:05:13.003 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:995: test_wait_for_osd: local dir=td/ceph-helpers 2024-09-16T06:05:13.003 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:996: test_wait_for_osd: setup td/ceph-helpers 2024-09-16T06:05:13.003 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T06:05:13.003 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T06:05:13.003 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T06:05:13.003 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T06:05:13.003 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T06:05:13.003 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:05:13.003 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:05:13.003 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:05:13.003 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:05:13.003 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:05:13.004 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:05:13.004 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:05:13.004 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:05:13.004 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:05:13.005 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:05:13.005 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:05:13.005 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:05:13.006 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:05:13.006 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:05:13.006 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:05:13.006 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:05:13.007 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:05:13.008 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T06:05:13.008 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T06:05:13.009 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:05:13.009 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:05:13.009 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:05:13.009 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T06:05:13.010 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:05:13.010 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:05:13.010 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T06:05:13.011 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T06:05:13.011 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:05:13.011 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:05:13.011 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T06:05:13.013 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T06:05:13.013 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T06:05:13.013 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T06:05:13.013 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T06:05:13.014 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:997: test_wait_for_osd: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T06:05:13.014 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T06:05:13.014 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T06:05:13.014 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T06:05:13.014 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T06:05:13.014 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T06:05:13.014 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T06:05:13.414 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T06:05:13.415 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:05:13.415 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:05:13.415 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:05:13.415 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:05:13.415 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:05:13.415 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:05:13.415 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T06:05:13.582 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T06:05:13.583 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T06:05:13.583 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T06:05:13.583 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T06:05:13.583 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T06:05:13.583 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T06:05:13.583 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T06:05:13.583 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T06:05:13.583 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T06:05:13.584 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:05:13.584 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:05:13.584 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:05:13.584 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T06:05:13.584 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T06:05:13.584 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T06:05:13.664 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T06:05:13.664 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T06:05:13.664 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T06:05:13.664 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T06:05:13.664 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T06:05:13.665 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T06:05:13.665 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T06:05:13.665 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T06:05:13.665 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:05:13.665 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:05:13.665 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:05:13.665 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T06:05:13.665 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T06:05:13.665 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T06:05:13.742 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:998: test_wait_for_osd: run_mgr td/ceph-helpers x 2024-09-16T06:05:13.742 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T06:05:13.742 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T06:05:13.742 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T06:05:13.742 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T06:05:13.743 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T06:05:13.743 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T06:05:14.587 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T06:05:14.587 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:05:14.587 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:05:14.587 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:05:14.587 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:05:14.587 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:05:14.588 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:05:14.588 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T06:05:14.588 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T06:05:14.619 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:999: test_wait_for_osd: run_osd td/ceph-helpers 0 2024-09-16T06:05:14.619 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T06:05:14.619 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T06:05:14.619 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T06:05:14.619 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T06:05:14.620 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T06:05:14.620 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T06:05:14.620 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:05:14.620 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:05:14.620 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:05:14.620 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T06:05:14.620 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T06:05:14.620 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T06:05:14.620 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T06:05:14.620 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T06:05:14.620 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T06:05:14.620 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:05:14.621 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:05:14.621 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:05:14.621 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:05:14.621 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:05:14.621 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:05:14.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:05:14.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:05:14.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T06:05:14.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T06:05:14.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T06:05:14.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T06:05:14.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:05:14.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:05:14.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:05:14.622 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:05:14.622 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T06:05:14.622 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T06:05:14.622 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T06:05:14.622 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T06:05:14.622 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c31f2022-d9f8-4a75-a45a-e4d10b385043 2024-09-16T06:05:14.622 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c31f2022-d9f8-4a75-a45a-e4d10b385043' 2024-09-16T06:05:14.623 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 c31f2022-d9f8-4a75-a45a-e4d10b385043 2024-09-16T06:05:14.623 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T06:05:14.639 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCayudmZq4PJhAAA7lOY7qtaJ8LnnIYAMCfMQ== 2024-09-16T06:05:14.639 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCayudmZq4PJhAAA7lOY7qtaJ8LnnIYAMCfMQ=="}' 2024-09-16T06:05:14.639 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c31f2022-d9f8-4a75-a45a-e4d10b385043 -i td/ceph-helpers/0/new.json 2024-09-16T06:05:14.907 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:05:14.917 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T06:05:14.918 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCayudmZq4PJhAAA7lOY7qtaJ8LnnIYAMCfMQ== --osd-uuid c31f2022-d9f8-4a75-a45a-e4d10b385043 2024-09-16T06:05:14.948 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:05:14.947+0000 7f179dcb4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:05:15.001 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:05:15.000+0000 7f179dcb4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:05:15.002 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:05:15.001+0000 7f179dcb4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:05:15.002 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:05:15.002+0000 7f179dcb4640 -1 bdev(0x55d90d049c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-16T06:05:15.002 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:05:15.002+0000 7f179dcb4640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T06:05:20.498 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T06:05:20.498 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T06:05:20.499 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T06:05:20.499 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T06:05:20.499 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T06:05:21.013 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T06:05:21.014 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T06:05:21.014 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T06:05:21.014 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T06:05:21.015 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T06:05:21.015 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T06:05:21.040 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:05:21.039+0000 7f64af415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:05:21.071 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:05:21.070+0000 7f64af415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:05:21.072 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:05:21.071+0000 7f64af415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:05:21.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T06:05:21.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:05:21.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T06:05:21.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:05:21.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:05:21.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:05:21.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:05:21.419 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:05:21.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:05:21.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:05:21.827 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:05:22.178 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:05:22.177+0000 7f64af415640 -1 Falling back to public interface 2024-09-16T06:05:22.828 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:05:22.829 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:05:22.829 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:05:22.830 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T06:05:22.830 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:05:22.830 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:05:23.236 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:05:24.237 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:05:24.237 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:05:24.238 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:05:24.239 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T06:05:24.239 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:05:24.239 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:05:24.619 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:05:24.991 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:05:24.990+0000 7f64af415640 -1 osd.0 0 log_to_monitors true 2024-09-16T06:05:25.620 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:05:25.620 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:05:25.621 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T06:05:25.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T06:05:25.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:05:25.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:05:26.131 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:05:27.133 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:05:27.133 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:05:27.133 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T06:05:27.134 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T06:05:27.134 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:05:27.134 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:05:27.550 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:05:28.552 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:05:28.552 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:05:28.552 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T06:05:28.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T06:05:28.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:05:28.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:05:28.998 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:05:29.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:05:29.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:05:29.999 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T06:05:30.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T06:05:30.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:05:30.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:05:30.380 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:05:31.382 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:05:31.382 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:05:31.382 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T06:05:31.383 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T06:05:31.383 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:05:31.383 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:05:31.901 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:05:32.902 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:05:32.902 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:05:32.902 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T06:05:32.903 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T06:05:32.903 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:05:32.903 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:05:33.318 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:05:34.320 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:05:34.320 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:05:34.320 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T06:05:34.321 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T06:05:34.321 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:05:34.321 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:05:34.802 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:05:35.804 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:05:35.804 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T06:05:35.805 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:05:35.805 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T06:05:35.805 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:05:35.805 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:05:36.197 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:05:37.199 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:05:37.199 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:05:37.199 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T06:05:37.200 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T06:05:37.200 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:05:37.200 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:05:37.639 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:05:38.640 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:05:38.640 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:05:38.641 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T06:05:38.641 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T06:05:38.641 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:05:38.641 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:05:39.478 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:05:40.479 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:05:40.479 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:05:40.479 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T06:05:40.480 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T06:05:40.480 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:05:40.480 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:05:40.954 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:05:41.955 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:05:41.955 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:05:41.956 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T06:05:41.956 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T06:05:41.956 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:05:41.956 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:05:42.697 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:05:43.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:05:43.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:05:43.699 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T06:05:43.699 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T06:05:43.699 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:05:43.699 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:05:44.465 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:05:45.466 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:05:45.466 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:05:45.466 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T06:05:45.467 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T06:05:45.467 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:05:45.467 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:05:46.279 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:05:47.280 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:05:47.280 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:05:47.280 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T06:05:47.281 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T06:05:47.281 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:05:47.281 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:05:47.958 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:05:48.960 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:05:48.960 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:05:48.961 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T06:05:48.961 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T06:05:48.961 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:05:48.961 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:05:49.385 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1035390668,v1:127.0.0.1:6803/1035390668] [v2:127.0.0.1:6804/1035390668,v1:127.0.0.1:6805/1035390668] exists,up c31f2022-d9f8-4a75-a45a-e4d10b385043 2024-09-16T06:05:49.389 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:05:49.389 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:05:49.389 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:05:49.389 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1000: test_wait_for_osd: run_osd td/ceph-helpers 1 2024-09-16T06:05:49.389 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T06:05:49.389 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T06:05:49.389 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T06:05:49.389 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T06:05:49.389 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-09-16T06:05:49.389 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T06:05:49.389 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:05:49.390 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:05:49.390 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:05:49.390 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-16T06:05:49.390 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-16T06:05:49.390 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T06:05:49.390 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T06:05:49.390 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T06:05:49.390 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T06:05:49.390 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:05:49.390 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:05:49.390 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:05:49.390 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:05:49.390 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:05:49.391 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:05:49.391 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:05:49.391 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:05:49.391 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T06:05:49.391 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T06:05:49.391 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T06:05:49.391 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T06:05:49.391 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:05:49.391 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:05:49.391 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:05:49.391 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:05:49.391 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T06:05:49.391 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T06:05:49.391 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-09-16T06:05:49.392 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T06:05:49.392 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=35c08ef8-2945-40e5-8766-7abd824d95a0 2024-09-16T06:05:49.392 INFO:tasks.workunit.client.0.smithi178.stdout:add osd1 35c08ef8-2945-40e5-8766-7abd824d95a0 2024-09-16T06:05:49.392 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 35c08ef8-2945-40e5-8766-7abd824d95a0' 2024-09-16T06:05:49.392 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T06:05:49.409 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC9yudmk6FXGBAA/8xfyFTAN+GiYvjprut/DQ== 2024-09-16T06:05:49.409 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC9yudmk6FXGBAA/8xfyFTAN+GiYvjprut/DQ=="}' 2024-09-16T06:05:49.409 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 35c08ef8-2945-40e5-8766-7abd824d95a0 -i td/ceph-helpers/1/new.json 2024-09-16T06:05:49.872 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:05:49.883 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-09-16T06:05:49.884 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC9yudmk6FXGBAA/8xfyFTAN+GiYvjprut/DQ== --osd-uuid 35c08ef8-2945-40e5-8766-7abd824d95a0 2024-09-16T06:05:49.913 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:05:49.912+0000 7fded3bd9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:05:49.917 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:05:49.917+0000 7fded3bd9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:05:49.918 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:05:49.918+0000 7fded3bd9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:05:49.919 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:05:49.918+0000 7fded3bd9640 -1 bdev(0x563cf3fe5c00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2024-09-16T06:05:49.919 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:05:49.919+0000 7fded3bd9640 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-16T06:05:55.910 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-16T06:05:55.910 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T06:05:55.911 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd1 key to auth repository 2024-09-16T06:05:55.911 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T06:05:55.911 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T06:05:56.424 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.1 2024-09-16T06:05:56.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T06:05:56.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T06:05:56.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T06:05:56.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T06:05:56.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T06:05:56.451 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:05:56.451+0000 7fc4cd9b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:05:56.478 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:05:56.477+0000 7fc4cd9b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:05:56.479 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:05:56.478+0000 7fc4cd9b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:05:56.895 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T06:05:56.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:05:56.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T06:05:56.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:05:56.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:05:56.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:05:56.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:05:56.896 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:05:56.897 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:05:56.897 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:05:57.309 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:05:57.332 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:05:57.332+0000 7fc4cd9b2640 -1 Falling back to public interface 2024-09-16T06:05:58.310 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:05:58.310 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:05:58.311 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:05:58.311 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T06:05:58.311 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:05:58.311 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:05:58.700 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:05:59.701 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:05:59.702 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:05:59.702 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:05:59.702 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T06:05:59.702 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:05:59.703 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:05:59.955 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:05:59.955+0000 7fc4cd9b2640 -1 osd.1 0 log_to_monitors true 2024-09-16T06:06:00.073 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:06:01.074 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:06:01.074 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:01.075 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T06:06:01.075 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T06:06:01.075 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:01.075 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:06:01.497 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:06:02.497 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:06:02.497 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:02.498 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T06:06:02.498 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T06:06:02.498 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:02.498 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:06:02.973 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:06:03.974 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:06:03.975 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:03.975 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T06:06:03.975 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T06:06:03.975 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:03.975 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:06:04.357 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:06:05.358 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:06:05.358 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:05.358 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T06:06:05.359 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T06:06:05.359 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:05.359 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:06:05.778 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:06:06.779 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:06:06.779 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:06.780 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T06:06:06.780 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T06:06:06.780 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:06.780 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:06:07.241 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:06:08.242 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:06:08.243 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:08.243 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T06:06:08.243 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T06:06:08.244 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:08.244 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:06:08.814 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:06:09.815 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:06:09.815 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:09.816 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T06:06:09.816 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T06:06:09.817 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:09.817 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:06:10.204 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:06:11.206 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:06:11.206 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:11.206 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T06:06:11.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T06:06:11.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:11.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:06:11.588 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:06:12.589 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:06:12.589 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:12.590 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T06:06:12.590 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T06:06:12.590 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:12.590 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:06:13.611 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:06:14.612 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:06:14.612 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:14.613 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T06:06:14.614 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T06:06:14.614 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:14.614 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:06:15.151 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:06:16.152 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:06:16.152 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:16.152 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T06:06:16.153 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T06:06:16.153 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:16.153 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:06:16.959 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:06:17.961 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:06:17.961 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:17.961 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T06:06:17.962 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T06:06:17.962 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:17.962 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:06:18.701 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:06:19.702 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:06:19.703 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:19.703 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T06:06:19.703 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T06:06:19.703 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:19.703 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:06:20.604 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:06:21.605 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:06:21.605 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:21.606 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T06:06:21.606 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T06:06:21.606 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:21.606 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:06:22.722 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:06:23.723 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:06:23.724 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:23.724 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T06:06:23.725 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T06:06:23.725 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:23.725 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:06:24.103 INFO:tasks.workunit.client.0.smithi178.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1726056813,v1:127.0.0.1:6811/1726056813] [v2:127.0.0.1:6812/1726056813,v1:127.0.0.1:6813/1726056813] exists,up 35c08ef8-2945-40e5-8766-7abd824d95a0 2024-09-16T06:06:24.104 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:06:24.104 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:06:24.104 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:06:24.104 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1001: test_wait_for_osd: wait_for_osd up 0 2024-09-16T06:06:24.104 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:06:24.105 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T06:06:24.105 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:06:24.105 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:06:24.105 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:24.105 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:06:24.105 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:06:24.106 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:24.106 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:06:24.524 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1035390668,v1:127.0.0.1:6803/1035390668] [v2:127.0.0.1:6804/1035390668,v1:127.0.0.1:6805/1035390668] exists,up c31f2022-d9f8-4a75-a45a-e4d10b385043 2024-09-16T06:06:24.526 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:06:24.526 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:06:24.526 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:06:24.526 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1002: test_wait_for_osd: wait_for_osd up 1 2024-09-16T06:06:24.526 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:06:24.526 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T06:06:24.526 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:06:24.526 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:06:24.526 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:24.526 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:06:24.526 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:06:24.527 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:24.527 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:06:24.940 INFO:tasks.workunit.client.0.smithi178.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1726056813,v1:127.0.0.1:6811/1726056813] [v2:127.0.0.1:6812/1726056813,v1:127.0.0.1:6813/1726056813] exists,up 35c08ef8-2945-40e5-8766-7abd824d95a0 2024-09-16T06:06:24.941 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:06:24.942 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:06:24.942 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:06:24.942 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1003: test_wait_for_osd: kill_daemons td/ceph-helpers TERM osd.0 2024-09-16T06:06:24.942 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:06:24.942 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:06:24.942 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:06:24.942 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:06:24.942 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:06:25.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:06:25.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1004: test_wait_for_osd: wait_for_osd down 0 2024-09-16T06:06:25.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2024-09-16T06:06:25.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T06:06:25.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:06:25.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:06:25.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:25.248 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:06:25.249 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:06:25.249 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:25.249 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2024-09-16T06:06:25.629 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 12 last_clean_interval [0,0) [v2:127.0.0.1:6802/1035390668,v1:127.0.0.1:6803/1035390668] [v2:127.0.0.1:6804/1035390668,v1:127.0.0.1:6805/1035390668] exists c31f2022-d9f8-4a75-a45a-e4d10b385043 2024-09-16T06:06:25.630 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:06:25.630 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:06:25.630 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:06:25.630 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1005: test_wait_for_osd: TIMEOUT=1 2024-09-16T06:06:25.631 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1005: test_wait_for_osd: wait_for_osd up 0 2024-09-16T06:06:25.631 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:06:25.631 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T06:06:25.631 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:06:25.631 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:06:25.631 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 1 )) 2024-09-16T06:06:25.631 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:06:25.631 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:06:25.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:25.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:06:26.136 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:06:27.138 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:06:27.138 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 1 )) 2024-09-16T06:06:27.138 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 1 2024-09-16T06:06:27.139 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1006: test_wait_for_osd: teardown td/ceph-helpers 2024-09-16T06:06:27.139 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T06:06:27.139 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T06:06:27.139 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T06:06:27.139 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:06:27.139 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:06:27.140 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:06:27.140 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:06:27.140 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:06:27.246 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:06:27.246 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:06:27.247 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:06:27.248 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:06:27.249 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:06:27.249 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:06:27.249 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:06:27.250 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:06:27.250 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:06:27.251 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:06:27.251 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:06:27.251 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:06:27.252 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T06:06:27.253 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T06:06:27.263 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:06:27.263 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:06:27.263 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:06:27.263 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T06:06:27.269 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:06:27.269 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:06:27.270 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T06:06:27.270 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_wait_for_peered td/ceph-helpers 2024-09-16T06:06:27.270 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1734: test_wait_for_peered: local dir=td/ceph-helpers 2024-09-16T06:06:27.270 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1736: test_wait_for_peered: setup td/ceph-helpers 2024-09-16T06:06:27.270 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T06:06:27.270 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T06:06:27.270 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T06:06:27.270 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T06:06:27.270 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T06:06:27.270 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:06:27.270 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:06:27.270 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:06:27.270 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:06:27.270 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:06:27.271 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:06:27.271 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:06:27.271 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:06:27.271 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:06:27.271 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:06:27.271 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:06:27.271 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:06:27.272 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:06:27.272 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:06:27.272 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:06:27.272 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:06:27.272 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:06:27.273 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T06:06:27.273 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T06:06:27.274 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:06:27.274 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:06:27.274 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:06:27.274 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T06:06:27.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:06:27.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:06:27.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T06:06:27.276 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T06:06:27.276 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:06:27.276 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:06:27.277 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T06:06:27.279 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T06:06:27.279 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T06:06:27.279 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T06:06:27.279 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T06:06:27.279 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1737: test_wait_for_peered: run_mon td/ceph-helpers a --osd_pool_default_size=2 2024-09-16T06:06:27.279 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T06:06:27.279 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T06:06:27.279 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T06:06:27.279 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T06:06:27.279 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T06:06:27.280 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2024-09-16T06:06:27.657 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T06:06:27.658 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:06:27.658 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:06:27.658 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:06:27.658 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:06:27.658 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:06:27.658 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:06:27.658 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-16T06:06:27.850 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T06:06:27.850 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T06:06:27.850 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T06:06:27.851 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T06:06:27.851 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T06:06:27.852 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T06:06:27.852 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T06:06:27.852 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T06:06:27.852 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T06:06:27.852 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:06:27.852 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:06:27.852 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:06:27.852 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T06:06:27.852 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T06:06:27.853 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T06:06:27.937 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T06:06:27.937 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T06:06:27.937 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T06:06:27.937 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T06:06:27.937 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T06:06:27.937 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T06:06:27.937 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T06:06:27.938 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T06:06:27.938 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:06:27.938 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:06:27.938 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:06:27.938 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T06:06:27.938 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T06:06:27.938 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T06:06:28.018 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1738: test_wait_for_peered: run_osd td/ceph-helpers 0 2024-09-16T06:06:28.018 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T06:06:28.018 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T06:06:28.018 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T06:06:28.018 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T06:06:28.018 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T06:06:28.019 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T06:06:28.019 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:06:28.019 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:06:28.019 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:06:28.019 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T06:06:28.019 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T06:06:28.019 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T06:06:28.019 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T06:06:28.019 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T06:06:28.019 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T06:06:28.019 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:06:28.019 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:06:28.020 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:06:28.020 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:06:28.020 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:06:28.020 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:06:28.020 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:06:28.020 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:06:28.020 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T06:06:28.020 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T06:06:28.020 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T06:06:28.020 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T06:06:28.020 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:06:28.020 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:06:28.020 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:06:28.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:06:28.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T06:06:28.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T06:06:28.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T06:06:28.021 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T06:06:28.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6600e4ea-35cc-44a8-9be1-2042cb0374cf 2024-09-16T06:06:28.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6600e4ea-35cc-44a8-9be1-2042cb0374cf' 2024-09-16T06:06:28.022 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 6600e4ea-35cc-44a8-9be1-2042cb0374cf 2024-09-16T06:06:28.022 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T06:06:28.037 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDkyudm03YvAhAAMVWxcOpHs27lVvcLt5WDZQ== 2024-09-16T06:06:28.037 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDkyudm03YvAhAAMVWxcOpHs27lVvcLt5WDZQ=="}' 2024-09-16T06:06:28.037 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6600e4ea-35cc-44a8-9be1-2042cb0374cf -i td/ceph-helpers/0/new.json 2024-09-16T06:06:28.890 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:06:28.899 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T06:06:28.900 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDkyudm03YvAhAAMVWxcOpHs27lVvcLt5WDZQ== --osd-uuid 6600e4ea-35cc-44a8-9be1-2042cb0374cf 2024-09-16T06:06:28.928 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:06:28.928+0000 7f385e29c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:06:28.958 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:06:28.958+0000 7f385e29c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:06:28.959 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:06:28.959+0000 7f385e29c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:06:28.959 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:06:28.959+0000 7f385e29c640 -1 bdev(0x561ef0c65c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-16T06:06:28.960 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:06:28.959+0000 7f385e29c640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T06:06:34.373 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T06:06:34.373 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T06:06:34.374 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T06:06:34.375 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T06:06:34.375 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T06:06:34.579 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T06:06:34.579 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T06:06:34.580 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T06:06:34.580 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T06:06:34.580 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T06:06:34.580 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T06:06:34.605 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:06:34.605+0000 7f1cdbe33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:06:34.615 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:06:34.614+0000 7f1cdbe33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:06:34.616 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:06:34.616+0000 7f1cdbe33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:06:34.788 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T06:06:34.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:06:34.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T06:06:34.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:06:34.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:06:34.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:34.789 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:06:34.790 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:06:34.790 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:34.790 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:06:34.998 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:06:35.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:06:35.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:36.000 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:06:36.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T06:06:36.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:36.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:06:36.191 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:06:36.220 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:06:36.220+0000 7f1cdbe33640 -1 Falling back to public interface 2024-09-16T06:06:37.192 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:06:37.192 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:37.193 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:06:37.193 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T06:06:37.193 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:37.193 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:06:37.386 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:06:38.387 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:06:38.387 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:38.388 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T06:06:38.388 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T06:06:38.388 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:38.388 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:06:38.573 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:06:38.957 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:06:38.956+0000 7f1cdbe33640 -1 osd.0 0 log_to_monitors true 2024-09-16T06:06:39.575 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:06:39.577 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:39.577 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T06:06:39.578 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T06:06:39.578 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:39.578 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:06:39.888 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:06:40.889 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:06:40.889 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:40.889 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T06:06:40.890 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T06:06:40.890 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:40.890 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:06:41.083 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:06:42.084 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:06:42.084 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:42.085 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T06:06:42.085 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T06:06:42.085 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:42.085 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:06:42.279 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:06:43.280 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:06:43.280 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:43.281 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T06:06:43.281 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T06:06:43.281 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:43.281 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:06:43.478 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:06:44.479 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:06:44.479 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:44.479 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T06:06:44.480 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T06:06:44.480 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:44.480 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:06:44.672 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:06:45.674 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:06:45.674 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:45.675 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T06:06:45.675 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T06:06:45.675 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:45.675 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:06:45.868 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:06:46.869 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:06:46.869 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:46.870 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T06:06:46.870 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T06:06:46.870 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:46.870 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:06:47.064 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:06:48.065 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:06:48.065 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:48.066 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T06:06:48.066 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T06:06:48.066 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:48.066 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:06:48.261 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:06:49.262 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:06:49.262 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:49.263 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T06:06:49.263 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T06:06:49.263 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:49.263 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:06:49.456 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:06:50.457 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:06:50.458 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:50.458 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T06:06:50.458 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T06:06:50.459 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:50.459 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:06:50.648 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:06:51.649 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:06:51.649 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:51.649 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T06:06:51.650 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T06:06:51.650 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:51.650 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:06:51.838 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:06:52.839 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:06:52.839 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:52.839 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T06:06:52.840 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T06:06:52.840 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:52.840 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:06:53.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:06:54.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:06:54.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:54.037 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T06:06:54.037 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T06:06:54.037 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:54.037 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:06:54.233 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:06:55.234 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:06:55.235 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:55.235 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T06:06:55.235 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T06:06:55.235 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:55.236 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:06:55.423 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:06:56.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:06:56.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:56.425 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T06:06:56.426 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T06:06:56.426 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:56.426 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:06:56.622 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:06:57.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:06:57.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:57.624 INFO:tasks.workunit.client.0.smithi178.stdout:19 2024-09-16T06:06:57.625 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-16T06:06:57.625 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:57.625 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:06:57.818 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:06:58.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:06:58.820 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:06:58.820 INFO:tasks.workunit.client.0.smithi178.stdout:20 2024-09-16T06:06:58.820 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-16T06:06:58.821 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:06:58.821 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:06:59.013 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:07:00.015 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:07:00.015 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:07:00.015 INFO:tasks.workunit.client.0.smithi178.stdout:21 2024-09-16T06:07:00.016 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-09-16T06:07:00.016 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:07:00.016 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:07:00.207 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2668697607,v1:127.0.0.1:6801/2668697607] [v2:127.0.0.1:6802/2668697607,v1:127.0.0.1:6803/2668697607] exists,up 6600e4ea-35cc-44a8-9be1-2042cb0374cf 2024-09-16T06:07:00.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:07:00.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:07:00.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:07:00.209 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1739: test_wait_for_peered: run_mgr td/ceph-helpers x 2024-09-16T06:07:00.209 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T06:07:00.209 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T06:07:00.209 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T06:07:00.209 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T06:07:00.209 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T06:07:00.209 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T06:07:00.480 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T06:07:00.480 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:07:00.480 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:07:00.480 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:07:00.480 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:07:00.481 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:07:00.481 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:07:00.481 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T06:07:00.481 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T06:07:00.509 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1740: test_wait_for_peered: create_rbd_pool 2024-09-16T06:07:00.509 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T06:07:00.726 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' does not exist 2024-09-16T06:07:00.736 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T06:07:00.736 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T06:07:01.317 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' already exists 2024-09-16T06:07:01.326 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T06:07:02.328 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T06:07:03.257 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1741: test_wait_for_peered: WAIT_FOR_CLEAN_TIMEOUT=1 2024-09-16T06:07:03.258 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1741: test_wait_for_peered: wait_for_clean 2024-09-16T06:07:03.258 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T06:07:03.258 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T06:07:03.258 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T06:07:03.258 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 1 .1 2024-09-16T06:07:03.258 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T06:07:03.258 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T06:07:03.258 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T06:07:03.258 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T06:07:03.258 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T06:07:03.283 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.3') 2024-09-16T06:07:03.283 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T06:07:03.283 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T06:07:03.283 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T06:07:03.284 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-16T06:07:03.284 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-16T06:07:03.509 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-09-16T06:07:03.509 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-16T06:07:03.509 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T06:07:03.509 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T06:07:03.650 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=0 2024-09-16T06:07:03.650 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 0 2024-09-16T06:07:03.650 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-0' 2024-09-16T06:07:03.650 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T06:07:03.651 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-0 2024-09-16T06:07:03.651 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T06:07:03.651 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-16T06:07:03.651 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-0 2024-09-16T06:07:03.652 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T06:07:03.652 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=0 2024-09-16T06:07:03.653 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.0 seq 0 2024-09-16T06:07:03.653 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 0' 2024-09-16T06:07:03.653 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T06:07:03.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 0 -lt 0 2024-09-16T06:07:03.843 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T06:07:03.843 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T06:07:03.843 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T06:07:04.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 == 0 2024-09-16T06:07:04.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: sleep 1 2024-09-16T06:07:05.057 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T06:07:05.140 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T06:07:05.140 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T06:07:05.570 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T06:07:05.570 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T06:07:05.570 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T06:07:05.570 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T06:07:05.570 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T06:07:05.571 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T06:07:05.571 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T06:07:05.571 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T06:07:05.986 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T06:07:05.987 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T06:07:05.987 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T06:07:05.987 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T06:07:06.486 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-16T06:07:06.486 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-16T06:07:06.486 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-16T06:07:06.486 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-16T06:07:06.486 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T06:07:06.486 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T06:07:06.586 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T06:07:06.587 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T06:07:06.587 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T06:07:06.588 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T06:07:06.588 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T06:07:06.588 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T06:07:06.588 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T06:07:06.588 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T06:07:06.954 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T06:07:06.955 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T06:07:06.955 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T06:07:06.955 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T06:07:07.459 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-16T06:07:07.459 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T06:07:07.459 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T06:07:07.459 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T06:07:07.460 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T06:07:07.460 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T06:07:07.460 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T06:07:07.460 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T06:07:07.460 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T06:07:07.962 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T06:07:07.962 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T06:07:07.963 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 4 )) 2024-09-16T06:07:07.963 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T06:07:07.963 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-16T06:07:08.164 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T06:07:08.164 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T06:07:08.164 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T06:07:08.164 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T06:07:08.164 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T06:07:08.164 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T06:07:08.164 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T06:07:08.165 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T06:07:08.526 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T06:07:08.526 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T06:07:08.526 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T06:07:08.527 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T06:07:09.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-16T06:07:09.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T06:07:09.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T06:07:09.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T06:07:09.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T06:07:09.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T06:07:09.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T06:07:09.031 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T06:07:09.031 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T06:07:09.576 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T06:07:09.576 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T06:07:09.576 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 4 )) 2024-09-16T06:07:09.576 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T06:07:09.576 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-16T06:07:09.977 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T06:07:09.978 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T06:07:09.978 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T06:07:09.978 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T06:07:09.978 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T06:07:09.978 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T06:07:09.978 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T06:07:09.978 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T06:07:10.344 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T06:07:10.344 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T06:07:10.344 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T06:07:10.344 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T06:07:10.844 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-16T06:07:10.845 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T06:07:10.845 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T06:07:10.845 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T06:07:10.845 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T06:07:10.845 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T06:07:10.845 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T06:07:10.845 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T06:07:10.845 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T06:07:11.356 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T06:07:11.356 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T06:07:11.356 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 4 )) 2024-09-16T06:07:11.356 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T06:07:11.356 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.3 2024-09-16T06:07:11.657 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T06:07:11.658 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T06:07:11.658 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T06:07:11.658 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T06:07:11.658 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T06:07:11.658 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T06:07:11.658 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T06:07:11.659 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T06:07:12.022 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T06:07:12.023 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T06:07:12.023 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T06:07:12.023 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T06:07:12.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-16T06:07:12.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T06:07:12.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T06:07:12.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T06:07:12.531 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T06:07:12.531 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T06:07:12.531 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T06:07:12.531 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T06:07:12.531 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T06:07:13.062 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T06:07:13.063 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T06:07:13.063 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 4 )) 2024-09-16T06:07:13.063 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-16T06:07:13.585 INFO:tasks.workunit.client.0.smithi178.stdout:{ 2024-09-16T06:07:13.585 INFO:tasks.workunit.client.0.smithi178.stdout: "cluster_fingerprint": "5f206ca1-d2f4-42cf-87aa-b34db3bba4e1", 2024-09-16T06:07:13.585 INFO:tasks.workunit.client.0.smithi178.stdout: "version": "19.1.1-326-g8921c8ef", 2024-09-16T06:07:13.585 INFO:tasks.workunit.client.0.smithi178.stdout: "commit": "8921c8ef947807b717db041d85f0761b7914d5fc", 2024-09-16T06:07:13.585 INFO:tasks.workunit.client.0.smithi178.stdout: "timestamp": "2024-09-16T06:07:13.584079+0000", 2024-09-16T06:07:13.585 INFO:tasks.workunit.client.0.smithi178.stdout: "tag": "", 2024-09-16T06:07:13.585 INFO:tasks.workunit.client.0.smithi178.stdout: "health": { 2024-09-16T06:07:13.585 INFO:tasks.workunit.client.0.smithi178.stdout: "status": "HEALTH_WARN", 2024-09-16T06:07:13.586 INFO:tasks.workunit.client.0.smithi178.stdout: "checks": { 2024-09-16T06:07:13.586 INFO:tasks.workunit.client.0.smithi178.stdout: "PG_DEGRADED": { 2024-09-16T06:07:13.586 INFO:tasks.workunit.client.0.smithi178.stdout: "severity": "HEALTH_WARN", 2024-09-16T06:07:13.586 INFO:tasks.workunit.client.0.smithi178.stdout: "summary": { 2024-09-16T06:07:13.586 INFO:tasks.workunit.client.0.smithi178.stdout: "message": "Degraded data redundancy: 2/4 objects degraded (50.000%), 2 pgs degraded", 2024-09-16T06:07:13.586 INFO:tasks.workunit.client.0.smithi178.stdout: "count": 2 2024-09-16T06:07:13.586 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.586 INFO:tasks.workunit.client.0.smithi178.stdout: "detail": [ 2024-09-16T06:07:13.586 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.586 INFO:tasks.workunit.client.0.smithi178.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0]" 2024-09-16T06:07:13.586 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.586 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.586 INFO:tasks.workunit.client.0.smithi178.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [0]" 2024-09-16T06:07:13.587 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:07:13.587 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:07:13.587 INFO:tasks.workunit.client.0.smithi178.stdout: "muted": false 2024-09-16T06:07:13.587 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.587 INFO:tasks.workunit.client.0.smithi178.stdout: "TOO_FEW_OSDS": { 2024-09-16T06:07:13.587 INFO:tasks.workunit.client.0.smithi178.stdout: "severity": "HEALTH_WARN", 2024-09-16T06:07:13.587 INFO:tasks.workunit.client.0.smithi178.stdout: "summary": { 2024-09-16T06:07:13.587 INFO:tasks.workunit.client.0.smithi178.stdout: "message": "OSD count 1 < osd_pool_default_size 3", 2024-09-16T06:07:13.587 INFO:tasks.workunit.client.0.smithi178.stdout: "count": 2 2024-09-16T06:07:13.587 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.587 INFO:tasks.workunit.client.0.smithi178.stdout: "detail": [], 2024-09-16T06:07:13.587 INFO:tasks.workunit.client.0.smithi178.stdout: "muted": false 2024-09-16T06:07:13.587 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:07:13.588 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.588 INFO:tasks.workunit.client.0.smithi178.stdout: "mutes": [] 2024-09-16T06:07:13.588 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.588 INFO:tasks.workunit.client.0.smithi178.stdout: "monmap_first_committed": 1, 2024-09-16T06:07:13.588 INFO:tasks.workunit.client.0.smithi178.stdout: "monmap_last_committed": 1, 2024-09-16T06:07:13.588 INFO:tasks.workunit.client.0.smithi178.stdout: "monmap": { 2024-09-16T06:07:13.588 INFO:tasks.workunit.client.0.smithi178.stdout: "epoch": 1, 2024-09-16T06:07:13.588 INFO:tasks.workunit.client.0.smithi178.stdout: "fsid": "e36fe3b7-e285-4199-9297-eafea05d044b", 2024-09-16T06:07:13.588 INFO:tasks.workunit.client.0.smithi178.stdout: "modified": "2024-09-16T06:06:27.302069Z", 2024-09-16T06:07:13.588 INFO:tasks.workunit.client.0.smithi178.stdout: "created": "2024-09-16T06:06:27.302069Z", 2024-09-16T06:07:13.588 INFO:tasks.workunit.client.0.smithi178.stdout: "min_mon_release": 19, 2024-09-16T06:07:13.588 INFO:tasks.workunit.client.0.smithi178.stdout: "min_mon_release_name": "squid", 2024-09-16T06:07:13.588 INFO:tasks.workunit.client.0.smithi178.stdout: "election_strategy": 1, 2024-09-16T06:07:13.589 INFO:tasks.workunit.client.0.smithi178.stdout: "disallowed_leaders: ": "", 2024-09-16T06:07:13.589 INFO:tasks.workunit.client.0.smithi178.stdout: "stretch_mode": false, 2024-09-16T06:07:13.589 INFO:tasks.workunit.client.0.smithi178.stdout: "tiebreaker_mon": "", 2024-09-16T06:07:13.589 INFO:tasks.workunit.client.0.smithi178.stdout: "removed_ranks: ": "", 2024-09-16T06:07:13.589 INFO:tasks.workunit.client.0.smithi178.stdout: "features": { 2024-09-16T06:07:13.589 INFO:tasks.workunit.client.0.smithi178.stdout: "persistent": [ 2024-09-16T06:07:13.589 INFO:tasks.workunit.client.0.smithi178.stdout: "kraken", 2024-09-16T06:07:13.589 INFO:tasks.workunit.client.0.smithi178.stdout: "luminous", 2024-09-16T06:07:13.589 INFO:tasks.workunit.client.0.smithi178.stdout: "mimic", 2024-09-16T06:07:13.589 INFO:tasks.workunit.client.0.smithi178.stdout: "osdmap-prune", 2024-09-16T06:07:13.589 INFO:tasks.workunit.client.0.smithi178.stdout: "nautilus", 2024-09-16T06:07:13.589 INFO:tasks.workunit.client.0.smithi178.stdout: "octopus", 2024-09-16T06:07:13.589 INFO:tasks.workunit.client.0.smithi178.stdout: "pacific", 2024-09-16T06:07:13.589 INFO:tasks.workunit.client.0.smithi178.stdout: "elector-pinging", 2024-09-16T06:07:13.590 INFO:tasks.workunit.client.0.smithi178.stdout: "quincy", 2024-09-16T06:07:13.590 INFO:tasks.workunit.client.0.smithi178.stdout: "reef", 2024-09-16T06:07:13.590 INFO:tasks.workunit.client.0.smithi178.stdout: "squid" 2024-09-16T06:07:13.590 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:07:13.590 INFO:tasks.workunit.client.0.smithi178.stdout: "optional": [] 2024-09-16T06:07:13.590 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.590 INFO:tasks.workunit.client.0.smithi178.stdout: "mons": [ 2024-09-16T06:07:13.590 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.590 INFO:tasks.workunit.client.0.smithi178.stdout: "rank": 0, 2024-09-16T06:07:13.590 INFO:tasks.workunit.client.0.smithi178.stdout: "name": "a", 2024-09-16T06:07:13.590 INFO:tasks.workunit.client.0.smithi178.stdout: "public_addrs": { 2024-09-16T06:07:13.590 INFO:tasks.workunit.client.0.smithi178.stdout: "addrvec": [ 2024-09-16T06:07:13.590 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.591 INFO:tasks.workunit.client.0.smithi178.stdout: "type": "v2", 2024-09-16T06:07:13.591 INFO:tasks.workunit.client.0.smithi178.stdout: "addr": "127.0.0.1:7109", 2024-09-16T06:07:13.591 INFO:tasks.workunit.client.0.smithi178.stdout: "nonce": 0 2024-09-16T06:07:13.591 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:07:13.591 INFO:tasks.workunit.client.0.smithi178.stdout: ] 2024-09-16T06:07:13.591 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.591 INFO:tasks.workunit.client.0.smithi178.stdout: "addr": "127.0.0.1:7109/0", 2024-09-16T06:07:13.591 INFO:tasks.workunit.client.0.smithi178.stdout: "public_addr": "127.0.0.1:7109/0", 2024-09-16T06:07:13.591 INFO:tasks.workunit.client.0.smithi178.stdout: "priority": 0, 2024-09-16T06:07:13.591 INFO:tasks.workunit.client.0.smithi178.stdout: "weight": 0, 2024-09-16T06:07:13.591 INFO:tasks.workunit.client.0.smithi178.stdout: "crush_location": "{}" 2024-09-16T06:07:13.591 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:07:13.591 INFO:tasks.workunit.client.0.smithi178.stdout: ] 2024-09-16T06:07:13.591 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.592 INFO:tasks.workunit.client.0.smithi178.stdout: "quorum": [ 2024-09-16T06:07:13.592 INFO:tasks.workunit.client.0.smithi178.stdout: 0 2024-09-16T06:07:13.592 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:07:13.592 INFO:tasks.workunit.client.0.smithi178.stdout: "osdmap": { 2024-09-16T06:07:13.592 INFO:tasks.workunit.client.0.smithi178.stdout: "epoch": 12, 2024-09-16T06:07:13.592 INFO:tasks.workunit.client.0.smithi178.stdout: "fsid": "e36fe3b7-e285-4199-9297-eafea05d044b", 2024-09-16T06:07:13.592 INFO:tasks.workunit.client.0.smithi178.stdout: "created": "2024-09-16T06:06:28.104728+0000", 2024-09-16T06:07:13.592 INFO:tasks.workunit.client.0.smithi178.stdout: "modified": "2024-09-16T06:07:02.921104+0000", 2024-09-16T06:07:13.592 INFO:tasks.workunit.client.0.smithi178.stdout: "last_up_change": "2024-09-16T06:06:59.929050+0000", 2024-09-16T06:07:13.592 INFO:tasks.workunit.client.0.smithi178.stdout: "last_in_change": "2024-09-16T06:06:28.790044+0000", 2024-09-16T06:07:13.592 INFO:tasks.workunit.client.0.smithi178.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T06:07:13.592 INFO:tasks.workunit.client.0.smithi178.stdout: "flags_num": 5799936, 2024-09-16T06:07:13.592 INFO:tasks.workunit.client.0.smithi178.stdout: "flags_set": [ 2024-09-16T06:07:13.593 INFO:tasks.workunit.client.0.smithi178.stdout: "pglog_hardlimit", 2024-09-16T06:07:13.593 INFO:tasks.workunit.client.0.smithi178.stdout: "purged_snapdirs", 2024-09-16T06:07:13.593 INFO:tasks.workunit.client.0.smithi178.stdout: "recovery_deletes", 2024-09-16T06:07:13.593 INFO:tasks.workunit.client.0.smithi178.stdout: "sortbitwise" 2024-09-16T06:07:13.593 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:07:13.593 INFO:tasks.workunit.client.0.smithi178.stdout: "crush_version": 3, 2024-09-16T06:07:13.593 INFO:tasks.workunit.client.0.smithi178.stdout: "full_ratio": 0.99000000953674316, 2024-09-16T06:07:13.593 INFO:tasks.workunit.client.0.smithi178.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-16T06:07:13.593 INFO:tasks.workunit.client.0.smithi178.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T06:07:13.593 INFO:tasks.workunit.client.0.smithi178.stdout: "cluster_snapshot": "", 2024-09-16T06:07:13.593 INFO:tasks.workunit.client.0.smithi178.stdout: "pool_max": 1, 2024-09-16T06:07:13.593 INFO:tasks.workunit.client.0.smithi178.stdout: "max_osd": 1, 2024-09-16T06:07:13.593 INFO:tasks.workunit.client.0.smithi178.stdout: "require_min_compat_client": "luminous", 2024-09-16T06:07:13.594 INFO:tasks.workunit.client.0.smithi178.stdout: "min_compat_client": "jewel", 2024-09-16T06:07:13.594 INFO:tasks.workunit.client.0.smithi178.stdout: "require_osd_release": "squid", 2024-09-16T06:07:13.594 INFO:tasks.workunit.client.0.smithi178.stdout: "allow_crimson": false, 2024-09-16T06:07:13.594 INFO:tasks.workunit.client.0.smithi178.stdout: "pools": [ 2024-09-16T06:07:13.594 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.594 INFO:tasks.workunit.client.0.smithi178.stdout: "pool": 1, 2024-09-16T06:07:13.594 INFO:tasks.workunit.client.0.smithi178.stdout: "pool_name": "rbd", 2024-09-16T06:07:13.594 INFO:tasks.workunit.client.0.smithi178.stdout: "create_time": "2024-09-16T06:07:01.051990+0000", 2024-09-16T06:07:13.594 INFO:tasks.workunit.client.0.smithi178.stdout: "flags": 8193, 2024-09-16T06:07:13.594 INFO:tasks.workunit.client.0.smithi178.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T06:07:13.594 INFO:tasks.workunit.client.0.smithi178.stdout: "type": 1, 2024-09-16T06:07:13.594 INFO:tasks.workunit.client.0.smithi178.stdout: "size": 2, 2024-09-16T06:07:13.594 INFO:tasks.workunit.client.0.smithi178.stdout: "min_size": 1, 2024-09-16T06:07:13.595 INFO:tasks.workunit.client.0.smithi178.stdout: "crush_rule": 0, 2024-09-16T06:07:13.595 INFO:tasks.workunit.client.0.smithi178.stdout: "peering_crush_bucket_count": 0, 2024-09-16T06:07:13.595 INFO:tasks.workunit.client.0.smithi178.stdout: "peering_crush_bucket_target": 0, 2024-09-16T06:07:13.595 INFO:tasks.workunit.client.0.smithi178.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T06:07:13.595 INFO:tasks.workunit.client.0.smithi178.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T06:07:13.595 INFO:tasks.workunit.client.0.smithi178.stdout: "is_stretch_pool": false, 2024-09-16T06:07:13.595 INFO:tasks.workunit.client.0.smithi178.stdout: "object_hash": 2, 2024-09-16T06:07:13.595 INFO:tasks.workunit.client.0.smithi178.stdout: "pg_autoscale_mode": "off", 2024-09-16T06:07:13.595 INFO:tasks.workunit.client.0.smithi178.stdout: "pg_num": 4, 2024-09-16T06:07:13.595 INFO:tasks.workunit.client.0.smithi178.stdout: "pg_placement_num": 4, 2024-09-16T06:07:13.595 INFO:tasks.workunit.client.0.smithi178.stdout: "pg_placement_num_target": 4, 2024-09-16T06:07:13.595 INFO:tasks.workunit.client.0.smithi178.stdout: "pg_num_target": 4, 2024-09-16T06:07:13.596 INFO:tasks.workunit.client.0.smithi178.stdout: "pg_num_pending": 4, 2024-09-16T06:07:13.596 INFO:tasks.workunit.client.0.smithi178.stdout: "last_pg_merge_meta": { 2024-09-16T06:07:13.596 INFO:tasks.workunit.client.0.smithi178.stdout: "source_pgid": "0.0", 2024-09-16T06:07:13.596 INFO:tasks.workunit.client.0.smithi178.stdout: "ready_epoch": 0, 2024-09-16T06:07:13.596 INFO:tasks.workunit.client.0.smithi178.stdout: "last_epoch_started": 0, 2024-09-16T06:07:13.596 INFO:tasks.workunit.client.0.smithi178.stdout: "last_epoch_clean": 0, 2024-09-16T06:07:13.596 INFO:tasks.workunit.client.0.smithi178.stdout: "source_version": "0'0", 2024-09-16T06:07:13.597 INFO:tasks.workunit.client.0.smithi178.stdout: "target_version": "0'0" 2024-09-16T06:07:13.597 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.597 INFO:tasks.workunit.client.0.smithi178.stdout: "last_change": "12", 2024-09-16T06:07:13.597 INFO:tasks.workunit.client.0.smithi178.stdout: "last_force_op_resend": "0", 2024-09-16T06:07:13.597 INFO:tasks.workunit.client.0.smithi178.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T06:07:13.597 INFO:tasks.workunit.client.0.smithi178.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T06:07:13.597 INFO:tasks.workunit.client.0.smithi178.stdout: "auid": 0, 2024-09-16T06:07:13.597 INFO:tasks.workunit.client.0.smithi178.stdout: "snap_mode": "selfmanaged", 2024-09-16T06:07:13.597 INFO:tasks.workunit.client.0.smithi178.stdout: "snap_seq": 2, 2024-09-16T06:07:13.597 INFO:tasks.workunit.client.0.smithi178.stdout: "snap_epoch": 12, 2024-09-16T06:07:13.597 INFO:tasks.workunit.client.0.smithi178.stdout: "pool_snaps": [], 2024-09-16T06:07:13.597 INFO:tasks.workunit.client.0.smithi178.stdout: "removed_snaps": "[]", 2024-09-16T06:07:13.598 INFO:tasks.workunit.client.0.smithi178.stdout: "quota_max_bytes": 0, 2024-09-16T06:07:13.598 INFO:tasks.workunit.client.0.smithi178.stdout: "quota_max_objects": 0, 2024-09-16T06:07:13.598 INFO:tasks.workunit.client.0.smithi178.stdout: "tiers": [], 2024-09-16T06:07:13.598 INFO:tasks.workunit.client.0.smithi178.stdout: "tier_of": -1, 2024-09-16T06:07:13.598 INFO:tasks.workunit.client.0.smithi178.stdout: "read_tier": -1, 2024-09-16T06:07:13.598 INFO:tasks.workunit.client.0.smithi178.stdout: "write_tier": -1, 2024-09-16T06:07:13.598 INFO:tasks.workunit.client.0.smithi178.stdout: "cache_mode": "none", 2024-09-16T06:07:13.598 INFO:tasks.workunit.client.0.smithi178.stdout: "target_max_bytes": 0, 2024-09-16T06:07:13.598 INFO:tasks.workunit.client.0.smithi178.stdout: "target_max_objects": 0, 2024-09-16T06:07:13.598 INFO:tasks.workunit.client.0.smithi178.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T06:07:13.598 INFO:tasks.workunit.client.0.smithi178.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T06:07:13.598 INFO:tasks.workunit.client.0.smithi178.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T06:07:13.599 INFO:tasks.workunit.client.0.smithi178.stdout: "cache_min_flush_age": 0, 2024-09-16T06:07:13.599 INFO:tasks.workunit.client.0.smithi178.stdout: "cache_min_evict_age": 0, 2024-09-16T06:07:13.599 INFO:tasks.workunit.client.0.smithi178.stdout: "erasure_code_profile": "", 2024-09-16T06:07:13.599 INFO:tasks.workunit.client.0.smithi178.stdout: "hit_set_params": { 2024-09-16T06:07:13.599 INFO:tasks.workunit.client.0.smithi178.stdout: "type": "none" 2024-09-16T06:07:13.599 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.599 INFO:tasks.workunit.client.0.smithi178.stdout: "hit_set_period": 0, 2024-09-16T06:07:13.599 INFO:tasks.workunit.client.0.smithi178.stdout: "hit_set_count": 0, 2024-09-16T06:07:13.599 INFO:tasks.workunit.client.0.smithi178.stdout: "use_gmt_hitset": true, 2024-09-16T06:07:13.599 INFO:tasks.workunit.client.0.smithi178.stdout: "min_read_recency_for_promote": 0, 2024-09-16T06:07:13.599 INFO:tasks.workunit.client.0.smithi178.stdout: "min_write_recency_for_promote": 0, 2024-09-16T06:07:13.599 INFO:tasks.workunit.client.0.smithi178.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T06:07:13.599 INFO:tasks.workunit.client.0.smithi178.stdout: "hit_set_search_last_n": 0, 2024-09-16T06:07:13.599 INFO:tasks.workunit.client.0.smithi178.stdout: "grade_table": [], 2024-09-16T06:07:13.600 INFO:tasks.workunit.client.0.smithi178.stdout: "stripe_width": 0, 2024-09-16T06:07:13.600 INFO:tasks.workunit.client.0.smithi178.stdout: "expected_num_objects": 0, 2024-09-16T06:07:13.600 INFO:tasks.workunit.client.0.smithi178.stdout: "fast_read": false, 2024-09-16T06:07:13.600 INFO:tasks.workunit.client.0.smithi178.stdout: "options": {}, 2024-09-16T06:07:13.600 INFO:tasks.workunit.client.0.smithi178.stdout: "application_metadata": { 2024-09-16T06:07:13.600 INFO:tasks.workunit.client.0.smithi178.stdout: "rbd": {} 2024-09-16T06:07:13.600 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.600 INFO:tasks.workunit.client.0.smithi178.stdout: "read_balance": { 2024-09-16T06:07:13.600 INFO:tasks.workunit.client.0.smithi178.stdout: "score_type": "Fair distribution", 2024-09-16T06:07:13.600 INFO:tasks.workunit.client.0.smithi178.stdout: "score_acting": 1, 2024-09-16T06:07:13.600 INFO:tasks.workunit.client.0.smithi178.stdout: "score_stable": 1, 2024-09-16T06:07:13.600 INFO:tasks.workunit.client.0.smithi178.stdout: "optimal_score": 1, 2024-09-16T06:07:13.601 INFO:tasks.workunit.client.0.smithi178.stdout: "raw_score_acting": 1, 2024-09-16T06:07:13.601 INFO:tasks.workunit.client.0.smithi178.stdout: "raw_score_stable": 1, 2024-09-16T06:07:13.601 INFO:tasks.workunit.client.0.smithi178.stdout: "primary_affinity_weighted": 1, 2024-09-16T06:07:13.601 INFO:tasks.workunit.client.0.smithi178.stdout: "average_primary_affinity": 1, 2024-09-16T06:07:13.601 INFO:tasks.workunit.client.0.smithi178.stdout: "average_primary_affinity_weighted": 1 2024-09-16T06:07:13.601 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:07:13.601 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:07:13.601 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:07:13.601 INFO:tasks.workunit.client.0.smithi178.stdout: "osds": [ 2024-09-16T06:07:13.601 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.601 INFO:tasks.workunit.client.0.smithi178.stdout: "osd": 0, 2024-09-16T06:07:13.601 INFO:tasks.workunit.client.0.smithi178.stdout: "uuid": "6600e4ea-35cc-44a8-9be1-2042cb0374cf", 2024-09-16T06:07:13.601 INFO:tasks.workunit.client.0.smithi178.stdout: "up": 1, 2024-09-16T06:07:13.601 INFO:tasks.workunit.client.0.smithi178.stdout: "in": 1, 2024-09-16T06:07:13.601 INFO:tasks.workunit.client.0.smithi178.stdout: "weight": 1, 2024-09-16T06:07:13.602 INFO:tasks.workunit.client.0.smithi178.stdout: "primary_affinity": 1, 2024-09-16T06:07:13.602 INFO:tasks.workunit.client.0.smithi178.stdout: "last_clean_begin": 0, 2024-09-16T06:07:13.602 INFO:tasks.workunit.client.0.smithi178.stdout: "last_clean_end": 0, 2024-09-16T06:07:13.602 INFO:tasks.workunit.client.0.smithi178.stdout: "up_from": 5, 2024-09-16T06:07:13.602 INFO:tasks.workunit.client.0.smithi178.stdout: "up_thru": 7, 2024-09-16T06:07:13.602 INFO:tasks.workunit.client.0.smithi178.stdout: "down_at": 0, 2024-09-16T06:07:13.602 INFO:tasks.workunit.client.0.smithi178.stdout: "lost_at": 0, 2024-09-16T06:07:13.602 INFO:tasks.workunit.client.0.smithi178.stdout: "public_addrs": { 2024-09-16T06:07:13.602 INFO:tasks.workunit.client.0.smithi178.stdout: "addrvec": [ 2024-09-16T06:07:13.602 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.602 INFO:tasks.workunit.client.0.smithi178.stdout: "type": "v2", 2024-09-16T06:07:13.602 INFO:tasks.workunit.client.0.smithi178.stdout: "addr": "127.0.0.1:6800", 2024-09-16T06:07:13.602 INFO:tasks.workunit.client.0.smithi178.stdout: "nonce": 2668697607 2024-09-16T06:07:13.602 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.603 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.603 INFO:tasks.workunit.client.0.smithi178.stdout: "type": "v1", 2024-09-16T06:07:13.603 INFO:tasks.workunit.client.0.smithi178.stdout: "addr": "127.0.0.1:6801", 2024-09-16T06:07:13.603 INFO:tasks.workunit.client.0.smithi178.stdout: "nonce": 2668697607 2024-09-16T06:07:13.603 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:07:13.603 INFO:tasks.workunit.client.0.smithi178.stdout: ] 2024-09-16T06:07:13.603 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.603 INFO:tasks.workunit.client.0.smithi178.stdout: "cluster_addrs": { 2024-09-16T06:07:13.603 INFO:tasks.workunit.client.0.smithi178.stdout: "addrvec": [ 2024-09-16T06:07:13.603 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.603 INFO:tasks.workunit.client.0.smithi178.stdout: "type": "v2", 2024-09-16T06:07:13.603 INFO:tasks.workunit.client.0.smithi178.stdout: "addr": "127.0.0.1:6802", 2024-09-16T06:07:13.603 INFO:tasks.workunit.client.0.smithi178.stdout: "nonce": 2668697607 2024-09-16T06:07:13.604 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.604 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.604 INFO:tasks.workunit.client.0.smithi178.stdout: "type": "v1", 2024-09-16T06:07:13.604 INFO:tasks.workunit.client.0.smithi178.stdout: "addr": "127.0.0.1:6803", 2024-09-16T06:07:13.604 INFO:tasks.workunit.client.0.smithi178.stdout: "nonce": 2668697607 2024-09-16T06:07:13.604 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:07:13.604 INFO:tasks.workunit.client.0.smithi178.stdout: ] 2024-09-16T06:07:13.604 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.604 INFO:tasks.workunit.client.0.smithi178.stdout: "heartbeat_back_addrs": { 2024-09-16T06:07:13.604 INFO:tasks.workunit.client.0.smithi178.stdout: "addrvec": [ 2024-09-16T06:07:13.604 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.604 INFO:tasks.workunit.client.0.smithi178.stdout: "type": "v2", 2024-09-16T06:07:13.604 INFO:tasks.workunit.client.0.smithi178.stdout: "addr": "127.0.0.1:6806", 2024-09-16T06:07:13.604 INFO:tasks.workunit.client.0.smithi178.stdout: "nonce": 2668697607 2024-09-16T06:07:13.605 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.605 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.605 INFO:tasks.workunit.client.0.smithi178.stdout: "type": "v1", 2024-09-16T06:07:13.605 INFO:tasks.workunit.client.0.smithi178.stdout: "addr": "127.0.0.1:6807", 2024-09-16T06:07:13.605 INFO:tasks.workunit.client.0.smithi178.stdout: "nonce": 2668697607 2024-09-16T06:07:13.605 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:07:13.605 INFO:tasks.workunit.client.0.smithi178.stdout: ] 2024-09-16T06:07:13.605 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.605 INFO:tasks.workunit.client.0.smithi178.stdout: "heartbeat_front_addrs": { 2024-09-16T06:07:13.605 INFO:tasks.workunit.client.0.smithi178.stdout: "addrvec": [ 2024-09-16T06:07:13.605 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.605 INFO:tasks.workunit.client.0.smithi178.stdout: "type": "v2", 2024-09-16T06:07:13.605 INFO:tasks.workunit.client.0.smithi178.stdout: "addr": "127.0.0.1:6804", 2024-09-16T06:07:13.606 INFO:tasks.workunit.client.0.smithi178.stdout: "nonce": 2668697607 2024-09-16T06:07:13.606 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.606 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.606 INFO:tasks.workunit.client.0.smithi178.stdout: "type": "v1", 2024-09-16T06:07:13.606 INFO:tasks.workunit.client.0.smithi178.stdout: "addr": "127.0.0.1:6805", 2024-09-16T06:07:13.606 INFO:tasks.workunit.client.0.smithi178.stdout: "nonce": 2668697607 2024-09-16T06:07:13.606 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:07:13.606 INFO:tasks.workunit.client.0.smithi178.stdout: ] 2024-09-16T06:07:13.606 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.606 INFO:tasks.workunit.client.0.smithi178.stdout: "public_addr": "127.0.0.1:6801/2668697607", 2024-09-16T06:07:13.606 INFO:tasks.workunit.client.0.smithi178.stdout: "cluster_addr": "127.0.0.1:6803/2668697607", 2024-09-16T06:07:13.606 INFO:tasks.workunit.client.0.smithi178.stdout: "heartbeat_back_addr": "127.0.0.1:6807/2668697607", 2024-09-16T06:07:13.606 INFO:tasks.workunit.client.0.smithi178.stdout: "heartbeat_front_addr": "127.0.0.1:6805/2668697607", 2024-09-16T06:07:13.606 INFO:tasks.workunit.client.0.smithi178.stdout: "state": [ 2024-09-16T06:07:13.607 INFO:tasks.workunit.client.0.smithi178.stdout: "exists", 2024-09-16T06:07:13.607 INFO:tasks.workunit.client.0.smithi178.stdout: "up" 2024-09-16T06:07:13.607 INFO:tasks.workunit.client.0.smithi178.stdout: ] 2024-09-16T06:07:13.607 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:07:13.607 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:07:13.607 INFO:tasks.workunit.client.0.smithi178.stdout: "osd_xinfo": [ 2024-09-16T06:07:13.607 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.607 INFO:tasks.workunit.client.0.smithi178.stdout: "osd": 0, 2024-09-16T06:07:13.607 INFO:tasks.workunit.client.0.smithi178.stdout: "down_stamp": "0.000000", 2024-09-16T06:07:13.607 INFO:tasks.workunit.client.0.smithi178.stdout: "laggy_probability": 0, 2024-09-16T06:07:13.607 INFO:tasks.workunit.client.0.smithi178.stdout: "laggy_interval": 0, 2024-09-16T06:07:13.607 INFO:tasks.workunit.client.0.smithi178.stdout: "features": 4540701547738038271, 2024-09-16T06:07:13.608 INFO:tasks.workunit.client.0.smithi178.stdout: "old_weight": 0, 2024-09-16T06:07:13.608 INFO:tasks.workunit.client.0.smithi178.stdout: "last_purged_snaps_scrub": "2024-09-16T06:06:59.641093+0000", 2024-09-16T06:07:13.608 INFO:tasks.workunit.client.0.smithi178.stdout: "dead_epoch": 0 2024-09-16T06:07:13.608 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:07:13.608 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:07:13.608 INFO:tasks.workunit.client.0.smithi178.stdout: "pg_upmap": [], 2024-09-16T06:07:13.608 INFO:tasks.workunit.client.0.smithi178.stdout: "pg_upmap_items": [], 2024-09-16T06:07:13.608 INFO:tasks.workunit.client.0.smithi178.stdout: "pg_upmap_primaries": [], 2024-09-16T06:07:13.608 INFO:tasks.workunit.client.0.smithi178.stdout: "pg_temp": [], 2024-09-16T06:07:13.608 INFO:tasks.workunit.client.0.smithi178.stdout: "primary_temp": [], 2024-09-16T06:07:13.608 INFO:tasks.workunit.client.0.smithi178.stdout: "blocklist": {}, 2024-09-16T06:07:13.608 INFO:tasks.workunit.client.0.smithi178.stdout: "range_blocklist": {}, 2024-09-16T06:07:13.608 INFO:tasks.workunit.client.0.smithi178.stdout: "erasure_code_profiles": { 2024-09-16T06:07:13.609 INFO:tasks.workunit.client.0.smithi178.stdout: "default": { 2024-09-16T06:07:13.609 INFO:tasks.workunit.client.0.smithi178.stdout: "k": "2", 2024-09-16T06:07:13.609 INFO:tasks.workunit.client.0.smithi178.stdout: "m": "2", 2024-09-16T06:07:13.609 INFO:tasks.workunit.client.0.smithi178.stdout: "plugin": "jerasure", 2024-09-16T06:07:13.609 INFO:tasks.workunit.client.0.smithi178.stdout: "technique": "reed_sol_van" 2024-09-16T06:07:13.609 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:07:13.609 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.609 INFO:tasks.workunit.client.0.smithi178.stdout: "removed_snaps_queue": [ 2024-09-16T06:07:13.609 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.609 INFO:tasks.workunit.client.0.smithi178.stdout: "pool": 1, 2024-09-16T06:07:13.609 INFO:tasks.workunit.client.0.smithi178.stdout: "snaps": [ 2024-09-16T06:07:13.609 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.609 INFO:tasks.workunit.client.0.smithi178.stdout: "begin": 2, 2024-09-16T06:07:13.609 INFO:tasks.workunit.client.0.smithi178.stdout: "length": 1 2024-09-16T06:07:13.610 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:07:13.610 INFO:tasks.workunit.client.0.smithi178.stdout: ] 2024-09-16T06:07:13.610 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:07:13.610 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:07:13.610 INFO:tasks.workunit.client.0.smithi178.stdout: "new_removed_snaps": [ 2024-09-16T06:07:13.610 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.610 INFO:tasks.workunit.client.0.smithi178.stdout: "pool": 1, 2024-09-16T06:07:13.610 INFO:tasks.workunit.client.0.smithi178.stdout: "snaps": [ 2024-09-16T06:07:13.610 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.610 INFO:tasks.workunit.client.0.smithi178.stdout: "begin": 2, 2024-09-16T06:07:13.610 INFO:tasks.workunit.client.0.smithi178.stdout: "length": 1 2024-09-16T06:07:13.610 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:07:13.610 INFO:tasks.workunit.client.0.smithi178.stdout: ] 2024-09-16T06:07:13.610 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:07:13.611 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:07:13.611 INFO:tasks.workunit.client.0.smithi178.stdout: "new_purged_snaps": [], 2024-09-16T06:07:13.611 INFO:tasks.workunit.client.0.smithi178.stdout: "crush_node_flags": {}, 2024-09-16T06:07:13.611 INFO:tasks.workunit.client.0.smithi178.stdout: "device_class_flags": {}, 2024-09-16T06:07:13.611 INFO:tasks.workunit.client.0.smithi178.stdout: "stretch_mode": { 2024-09-16T06:07:13.611 INFO:tasks.workunit.client.0.smithi178.stdout: "stretch_mode_enabled": false, 2024-09-16T06:07:13.611 INFO:tasks.workunit.client.0.smithi178.stdout: "stretch_bucket_count": 0, 2024-09-16T06:07:13.611 INFO:tasks.workunit.client.0.smithi178.stdout: "degraded_stretch_mode": 0, 2024-09-16T06:07:13.611 INFO:tasks.workunit.client.0.smithi178.stdout: "recovering_stretch_mode": 0, 2024-09-16T06:07:13.611 INFO:tasks.workunit.client.0.smithi178.stdout: "stretch_mode_bucket": 0 2024-09-16T06:07:13.612 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:07:13.612 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.612 INFO:tasks.workunit.client.0.smithi178.stdout: "osd_metadata": [ 2024-09-16T06:07:13.612 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.612 INFO:tasks.workunit.client.0.smithi178.stdout: "id": 0, 2024-09-16T06:07:13.612 INFO:tasks.workunit.client.0.smithi178.stdout: "arch": "x86_64", 2024-09-16T06:07:13.612 INFO:tasks.workunit.client.0.smithi178.stdout: "back_addr": "[v2:127.0.0.1:6802/2668697607,v1:127.0.0.1:6803/2668697607]", 2024-09-16T06:07:13.612 INFO:tasks.workunit.client.0.smithi178.stdout: "back_iface": "", 2024-09-16T06:07:13.612 INFO:tasks.workunit.client.0.smithi178.stdout: "bluefs": "1", 2024-09-16T06:07:13.612 INFO:tasks.workunit.client.0.smithi178.stdout: "bluefs_dedicated_db": "0", 2024-09-16T06:07:13.612 INFO:tasks.workunit.client.0.smithi178.stdout: "bluefs_dedicated_wal": "0", 2024-09-16T06:07:13.612 INFO:tasks.workunit.client.0.smithi178.stdout: "bluefs_single_shared_device": "1", 2024-09-16T06:07:13.612 INFO:tasks.workunit.client.0.smithi178.stdout: "bluestore_allocation_from_file": "0", 2024-09-16T06:07:13.612 INFO:tasks.workunit.client.0.smithi178.stdout: "bluestore_bdev_access_mode": "file", 2024-09-16T06:07:13.613 INFO:tasks.workunit.client.0.smithi178.stdout: "bluestore_bdev_block_size": "4096", 2024-09-16T06:07:13.613 INFO:tasks.workunit.client.0.smithi178.stdout: "bluestore_bdev_devices": "sda", 2024-09-16T06:07:13.613 INFO:tasks.workunit.client.0.smithi178.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-16T06:07:13.613 INFO:tasks.workunit.client.0.smithi178.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-16T06:07:13.613 INFO:tasks.workunit.client.0.smithi178.stdout: "bluestore_bdev_path": "td/ceph-helpers/0/block", 2024-09-16T06:07:13.613 INFO:tasks.workunit.client.0.smithi178.stdout: "bluestore_bdev_rotational": "1", 2024-09-16T06:07:13.613 INFO:tasks.workunit.client.0.smithi178.stdout: "bluestore_bdev_size": "107374182400", 2024-09-16T06:07:13.613 INFO:tasks.workunit.client.0.smithi178.stdout: "bluestore_bdev_support_discard": "1", 2024-09-16T06:07:13.613 INFO:tasks.workunit.client.0.smithi178.stdout: "bluestore_bdev_type": "hdd", 2024-09-16T06:07:13.613 INFO:tasks.workunit.client.0.smithi178.stdout: "bluestore_min_alloc_size": "4096", 2024-09-16T06:07:13.613 INFO:tasks.workunit.client.0.smithi178.stdout: "ceph_release": "squid", 2024-09-16T06:07:13.613 INFO:tasks.workunit.client.0.smithi178.stdout: "ceph_version": "ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable)", 2024-09-16T06:07:13.613 INFO:tasks.workunit.client.0.smithi178.stdout: "ceph_version_short": "19.1.1-326-g8921c8ef", 2024-09-16T06:07:13.613 INFO:tasks.workunit.client.0.smithi178.stdout: "ceph_version_when_created": "ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable)", 2024-09-16T06:07:13.614 INFO:tasks.workunit.client.0.smithi178.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-16T06:07:13.614 INFO:tasks.workunit.client.0.smithi178.stdout: "created_at": "2024-09-16T06:06:33.747151Z", 2024-09-16T06:07:13.614 INFO:tasks.workunit.client.0.smithi178.stdout: "default_device_class": "hdd", 2024-09-16T06:07:13.614 INFO:tasks.workunit.client.0.smithi178.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XFJ4", 2024-09-16T06:07:13.614 INFO:tasks.workunit.client.0.smithi178.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-16T06:07:13.614 INFO:tasks.workunit.client.0.smithi178.stdout: "devices": "sda", 2024-09-16T06:07:13.614 INFO:tasks.workunit.client.0.smithi178.stdout: "distro": "centos", 2024-09-16T06:07:13.614 INFO:tasks.workunit.client.0.smithi178.stdout: "distro_description": "CentOS Stream 9", 2024-09-16T06:07:13.614 INFO:tasks.workunit.client.0.smithi178.stdout: "distro_version": "9", 2024-09-16T06:07:13.614 INFO:tasks.workunit.client.0.smithi178.stdout: "front_addr": "[v2:127.0.0.1:6800/2668697607,v1:127.0.0.1:6801/2668697607]", 2024-09-16T06:07:13.614 INFO:tasks.workunit.client.0.smithi178.stdout: "front_iface": "", 2024-09-16T06:07:13.614 INFO:tasks.workunit.client.0.smithi178.stdout: "hb_back_addr": "[v2:127.0.0.1:6806/2668697607,v1:127.0.0.1:6807/2668697607]", 2024-09-16T06:07:13.615 INFO:tasks.workunit.client.0.smithi178.stdout: "hb_front_addr": "[v2:127.0.0.1:6804/2668697607,v1:127.0.0.1:6805/2668697607]", 2024-09-16T06:07:13.615 INFO:tasks.workunit.client.0.smithi178.stdout: "hostname": "smithi178", 2024-09-16T06:07:13.615 INFO:tasks.workunit.client.0.smithi178.stdout: "journal_rotational": "1", 2024-09-16T06:07:13.615 INFO:tasks.workunit.client.0.smithi178.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-16T06:07:13.615 INFO:tasks.workunit.client.0.smithi178.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-16T06:07:13.615 INFO:tasks.workunit.client.0.smithi178.stdout: "mem_swap_kb": "0", 2024-09-16T06:07:13.615 INFO:tasks.workunit.client.0.smithi178.stdout: "mem_total_kb": "32487132", 2024-09-16T06:07:13.615 INFO:tasks.workunit.client.0.smithi178.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-16T06:07:13.615 INFO:tasks.workunit.client.0.smithi178.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-16T06:07:13.615 INFO:tasks.workunit.client.0.smithi178.stdout: "os": "Linux", 2024-09-16T06:07:13.615 INFO:tasks.workunit.client.0.smithi178.stdout: "osd_data": "td/ceph-helpers/0", 2024-09-16T06:07:13.615 INFO:tasks.workunit.client.0.smithi178.stdout: "osd_objectstore": "bluestore", 2024-09-16T06:07:13.615 INFO:tasks.workunit.client.0.smithi178.stdout: "osdspec_affinity": "", 2024-09-16T06:07:13.615 INFO:tasks.workunit.client.0.smithi178.stdout: "rotational": "1" 2024-09-16T06:07:13.616 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:07:13.616 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:07:13.616 INFO:tasks.workunit.client.0.smithi178.stdout: "osdmap_clean_epochs": { 2024-09-16T06:07:13.616 INFO:tasks.workunit.client.0.smithi178.stdout: "min_last_epoch_clean": 0, 2024-09-16T06:07:13.616 INFO:tasks.workunit.client.0.smithi178.stdout: "last_epoch_clean": { 2024-09-16T06:07:13.616 INFO:tasks.workunit.client.0.smithi178.stdout: "per_pool": [] 2024-09-16T06:07:13.616 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.616 INFO:tasks.workunit.client.0.smithi178.stdout: "osd_epochs": [ 2024-09-16T06:07:13.616 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.616 INFO:tasks.workunit.client.0.smithi178.stdout: "id": 0, 2024-09-16T06:07:13.616 INFO:tasks.workunit.client.0.smithi178.stdout: "epoch": 5 2024-09-16T06:07:13.616 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:07:13.617 INFO:tasks.workunit.client.0.smithi178.stdout: ] 2024-09-16T06:07:13.617 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.617 INFO:tasks.workunit.client.0.smithi178.stdout: "osdmap_first_committed": 1, 2024-09-16T06:07:13.617 INFO:tasks.workunit.client.0.smithi178.stdout: "osdmap_last_committed": 12, 2024-09-16T06:07:13.617 INFO:tasks.workunit.client.0.smithi178.stdout: "crushmap": { 2024-09-16T06:07:13.617 INFO:tasks.workunit.client.0.smithi178.stdout: "devices": [ 2024-09-16T06:07:13.617 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.617 INFO:tasks.workunit.client.0.smithi178.stdout: "id": 0, 2024-09-16T06:07:13.617 INFO:tasks.workunit.client.0.smithi178.stdout: "name": "osd.0", 2024-09-16T06:07:13.617 INFO:tasks.workunit.client.0.smithi178.stdout: "class": "hdd" 2024-09-16T06:07:13.617 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:07:13.617 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:07:13.617 INFO:tasks.workunit.client.0.smithi178.stdout: "types": [ 2024-09-16T06:07:13.618 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.618 INFO:tasks.workunit.client.0.smithi178.stdout: "type_id": 0, 2024-09-16T06:07:13.618 INFO:tasks.workunit.client.0.smithi178.stdout: "name": "osd" 2024-09-16T06:07:13.618 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.618 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.618 INFO:tasks.workunit.client.0.smithi178.stdout: "type_id": 1, 2024-09-16T06:07:13.618 INFO:tasks.workunit.client.0.smithi178.stdout: "name": "host" 2024-09-16T06:07:13.618 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.618 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.618 INFO:tasks.workunit.client.0.smithi178.stdout: "type_id": 2, 2024-09-16T06:07:13.618 INFO:tasks.workunit.client.0.smithi178.stdout: "name": "chassis" 2024-09-16T06:07:13.618 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.618 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.618 INFO:tasks.workunit.client.0.smithi178.stdout: "type_id": 3, 2024-09-16T06:07:13.619 INFO:tasks.workunit.client.0.smithi178.stdout: "name": "rack" 2024-09-16T06:07:13.619 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.619 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.619 INFO:tasks.workunit.client.0.smithi178.stdout: "type_id": 4, 2024-09-16T06:07:13.619 INFO:tasks.workunit.client.0.smithi178.stdout: "name": "row" 2024-09-16T06:07:13.619 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.619 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.619 INFO:tasks.workunit.client.0.smithi178.stdout: "type_id": 5, 2024-09-16T06:07:13.619 INFO:tasks.workunit.client.0.smithi178.stdout: "name": "pdu" 2024-09-16T06:07:13.619 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.619 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.619 INFO:tasks.workunit.client.0.smithi178.stdout: "type_id": 6, 2024-09-16T06:07:13.619 INFO:tasks.workunit.client.0.smithi178.stdout: "name": "pod" 2024-09-16T06:07:13.619 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.620 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.620 INFO:tasks.workunit.client.0.smithi178.stdout: "type_id": 7, 2024-09-16T06:07:13.620 INFO:tasks.workunit.client.0.smithi178.stdout: "name": "room" 2024-09-16T06:07:13.620 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.620 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.620 INFO:tasks.workunit.client.0.smithi178.stdout: "type_id": 8, 2024-09-16T06:07:13.620 INFO:tasks.workunit.client.0.smithi178.stdout: "name": "datacenter" 2024-09-16T06:07:13.620 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.620 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.620 INFO:tasks.workunit.client.0.smithi178.stdout: "type_id": 9, 2024-09-16T06:07:13.620 INFO:tasks.workunit.client.0.smithi178.stdout: "name": "zone" 2024-09-16T06:07:13.620 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.620 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.620 INFO:tasks.workunit.client.0.smithi178.stdout: "type_id": 10, 2024-09-16T06:07:13.620 INFO:tasks.workunit.client.0.smithi178.stdout: "name": "region" 2024-09-16T06:07:13.621 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.621 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.621 INFO:tasks.workunit.client.0.smithi178.stdout: "type_id": 11, 2024-09-16T06:07:13.621 INFO:tasks.workunit.client.0.smithi178.stdout: "name": "root" 2024-09-16T06:07:13.621 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:07:13.621 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:07:13.621 INFO:tasks.workunit.client.0.smithi178.stdout: "buckets": [ 2024-09-16T06:07:13.621 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.621 INFO:tasks.workunit.client.0.smithi178.stdout: "id": -1, 2024-09-16T06:07:13.621 INFO:tasks.workunit.client.0.smithi178.stdout: "name": "default", 2024-09-16T06:07:13.621 INFO:tasks.workunit.client.0.smithi178.stdout: "type_id": 11, 2024-09-16T06:07:13.621 INFO:tasks.workunit.client.0.smithi178.stdout: "type_name": "root", 2024-09-16T06:07:13.621 INFO:tasks.workunit.client.0.smithi178.stdout: "weight": 6402, 2024-09-16T06:07:13.621 INFO:tasks.workunit.client.0.smithi178.stdout: "alg": "straw2", 2024-09-16T06:07:13.622 INFO:tasks.workunit.client.0.smithi178.stdout: "hash": "rjenkins1", 2024-09-16T06:07:13.622 INFO:tasks.workunit.client.0.smithi178.stdout: "items": [ 2024-09-16T06:07:13.622 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.622 INFO:tasks.workunit.client.0.smithi178.stdout: "id": -3, 2024-09-16T06:07:13.622 INFO:tasks.workunit.client.0.smithi178.stdout: "weight": 6402, 2024-09-16T06:07:13.622 INFO:tasks.workunit.client.0.smithi178.stdout: "pos": 0 2024-09-16T06:07:13.622 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:07:13.622 INFO:tasks.workunit.client.0.smithi178.stdout: ] 2024-09-16T06:07:13.622 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.622 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.622 INFO:tasks.workunit.client.0.smithi178.stdout: "id": -2, 2024-09-16T06:07:13.622 INFO:tasks.workunit.client.0.smithi178.stdout: "name": "default~hdd", 2024-09-16T06:07:13.622 INFO:tasks.workunit.client.0.smithi178.stdout: "type_id": 11, 2024-09-16T06:07:13.623 INFO:tasks.workunit.client.0.smithi178.stdout: "type_name": "root", 2024-09-16T06:07:13.623 INFO:tasks.workunit.client.0.smithi178.stdout: "weight": 6402, 2024-09-16T06:07:13.623 INFO:tasks.workunit.client.0.smithi178.stdout: "alg": "straw2", 2024-09-16T06:07:13.623 INFO:tasks.workunit.client.0.smithi178.stdout: "hash": "rjenkins1", 2024-09-16T06:07:13.623 INFO:tasks.workunit.client.0.smithi178.stdout: "items": [ 2024-09-16T06:07:13.623 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.623 INFO:tasks.workunit.client.0.smithi178.stdout: "id": -4, 2024-09-16T06:07:13.623 INFO:tasks.workunit.client.0.smithi178.stdout: "weight": 6402, 2024-09-16T06:07:13.627 INFO:tasks.workunit.client.0.smithi178.stdout: "pos": 0 2024-09-16T06:07:13.627 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:07:13.627 INFO:tasks.workunit.client.0.smithi178.stdout: ] 2024-09-16T06:07:13.627 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.627 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.627 INFO:tasks.workunit.client.0.smithi178.stdout: "id": -3, 2024-09-16T06:07:13.627 INFO:tasks.workunit.client.0.smithi178.stdout: "name": "smithi178", 2024-09-16T06:07:13.628 INFO:tasks.workunit.client.0.smithi178.stdout: "type_id": 1, 2024-09-16T06:07:13.628 INFO:tasks.workunit.client.0.smithi178.stdout: "type_name": "host", 2024-09-16T06:07:13.628 INFO:tasks.workunit.client.0.smithi178.stdout: "weight": 6402, 2024-09-16T06:07:13.628 INFO:tasks.workunit.client.0.smithi178.stdout: "alg": "straw2", 2024-09-16T06:07:13.628 INFO:tasks.workunit.client.0.smithi178.stdout: "hash": "rjenkins1", 2024-09-16T06:07:13.628 INFO:tasks.workunit.client.0.smithi178.stdout: "items": [ 2024-09-16T06:07:13.628 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.628 INFO:tasks.workunit.client.0.smithi178.stdout: "id": 0, 2024-09-16T06:07:13.628 INFO:tasks.workunit.client.0.smithi178.stdout: "weight": 6402, 2024-09-16T06:07:13.628 INFO:tasks.workunit.client.0.smithi178.stdout: "pos": 0 2024-09-16T06:07:13.628 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:07:13.628 INFO:tasks.workunit.client.0.smithi178.stdout: ] 2024-09-16T06:07:13.629 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.629 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.629 INFO:tasks.workunit.client.0.smithi178.stdout: "id": -4, 2024-09-16T06:07:13.629 INFO:tasks.workunit.client.0.smithi178.stdout: "name": "smithi178~hdd", 2024-09-16T06:07:13.629 INFO:tasks.workunit.client.0.smithi178.stdout: "type_id": 1, 2024-09-16T06:07:13.629 INFO:tasks.workunit.client.0.smithi178.stdout: "type_name": "host", 2024-09-16T06:07:13.629 INFO:tasks.workunit.client.0.smithi178.stdout: "weight": 6402, 2024-09-16T06:07:13.629 INFO:tasks.workunit.client.0.smithi178.stdout: "alg": "straw2", 2024-09-16T06:07:13.629 INFO:tasks.workunit.client.0.smithi178.stdout: "hash": "rjenkins1", 2024-09-16T06:07:13.629 INFO:tasks.workunit.client.0.smithi178.stdout: "items": [ 2024-09-16T06:07:13.629 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.629 INFO:tasks.workunit.client.0.smithi178.stdout: "id": 0, 2024-09-16T06:07:13.629 INFO:tasks.workunit.client.0.smithi178.stdout: "weight": 6402, 2024-09-16T06:07:13.629 INFO:tasks.workunit.client.0.smithi178.stdout: "pos": 0 2024-09-16T06:07:13.630 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:07:13.630 INFO:tasks.workunit.client.0.smithi178.stdout: ] 2024-09-16T06:07:13.630 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:07:13.630 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:07:13.630 INFO:tasks.workunit.client.0.smithi178.stdout: "rules": [ 2024-09-16T06:07:13.630 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.630 INFO:tasks.workunit.client.0.smithi178.stdout: "rule_id": 0, 2024-09-16T06:07:13.630 INFO:tasks.workunit.client.0.smithi178.stdout: "rule_name": "replicated_rule", 2024-09-16T06:07:13.630 INFO:tasks.workunit.client.0.smithi178.stdout: "type": 1, 2024-09-16T06:07:13.630 INFO:tasks.workunit.client.0.smithi178.stdout: "steps": [ 2024-09-16T06:07:13.630 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.630 INFO:tasks.workunit.client.0.smithi178.stdout: "op": "take", 2024-09-16T06:07:13.630 INFO:tasks.workunit.client.0.smithi178.stdout: "item": -1, 2024-09-16T06:07:13.630 INFO:tasks.workunit.client.0.smithi178.stdout: "item_name": "default" 2024-09-16T06:07:13.630 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.631 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.631 INFO:tasks.workunit.client.0.smithi178.stdout: "op": "choose_firstn", 2024-09-16T06:07:13.631 INFO:tasks.workunit.client.0.smithi178.stdout: "num": 0, 2024-09-16T06:07:13.631 INFO:tasks.workunit.client.0.smithi178.stdout: "type": "osd" 2024-09-16T06:07:13.631 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.631 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.631 INFO:tasks.workunit.client.0.smithi178.stdout: "op": "emit" 2024-09-16T06:07:13.631 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:07:13.631 INFO:tasks.workunit.client.0.smithi178.stdout: ] 2024-09-16T06:07:13.631 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:07:13.631 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:07:13.631 INFO:tasks.workunit.client.0.smithi178.stdout: "tunables": { 2024-09-16T06:07:13.631 INFO:tasks.workunit.client.0.smithi178.stdout: "choose_local_tries": 0, 2024-09-16T06:07:13.631 INFO:tasks.workunit.client.0.smithi178.stdout: "choose_local_fallback_tries": 0, 2024-09-16T06:07:13.632 INFO:tasks.workunit.client.0.smithi178.stdout: "choose_total_tries": 50, 2024-09-16T06:07:13.632 INFO:tasks.workunit.client.0.smithi178.stdout: "chooseleaf_descend_once": 1, 2024-09-16T06:07:13.632 INFO:tasks.workunit.client.0.smithi178.stdout: "chooseleaf_vary_r": 1, 2024-09-16T06:07:13.632 INFO:tasks.workunit.client.0.smithi178.stdout: "chooseleaf_stable": 1, 2024-09-16T06:07:13.632 INFO:tasks.workunit.client.0.smithi178.stdout: "msr_descents": 100, 2024-09-16T06:07:13.632 INFO:tasks.workunit.client.0.smithi178.stdout: "msr_collision_tries": 100, 2024-09-16T06:07:13.632 INFO:tasks.workunit.client.0.smithi178.stdout: "straw_calc_version": 1, 2024-09-16T06:07:13.632 INFO:tasks.workunit.client.0.smithi178.stdout: "allowed_bucket_algs": 54, 2024-09-16T06:07:13.632 INFO:tasks.workunit.client.0.smithi178.stdout: "profile": "jewel", 2024-09-16T06:07:13.632 INFO:tasks.workunit.client.0.smithi178.stdout: "optimal_tunables": 1, 2024-09-16T06:07:13.632 INFO:tasks.workunit.client.0.smithi178.stdout: "legacy_tunables": 0, 2024-09-16T06:07:13.632 INFO:tasks.workunit.client.0.smithi178.stdout: "minimum_required_version": "jewel", 2024-09-16T06:07:13.632 INFO:tasks.workunit.client.0.smithi178.stdout: "require_feature_tunables": 1, 2024-09-16T06:07:13.632 INFO:tasks.workunit.client.0.smithi178.stdout: "require_feature_tunables2": 1, 2024-09-16T06:07:13.633 INFO:tasks.workunit.client.0.smithi178.stdout: "has_v2_rules": 0, 2024-09-16T06:07:13.633 INFO:tasks.workunit.client.0.smithi178.stdout: "require_feature_tunables3": 1, 2024-09-16T06:07:13.633 INFO:tasks.workunit.client.0.smithi178.stdout: "has_v3_rules": 0, 2024-09-16T06:07:13.633 INFO:tasks.workunit.client.0.smithi178.stdout: "has_v4_buckets": 1, 2024-09-16T06:07:13.633 INFO:tasks.workunit.client.0.smithi178.stdout: "require_feature_tunables5": 1, 2024-09-16T06:07:13.633 INFO:tasks.workunit.client.0.smithi178.stdout: "has_v5_rules": 0, 2024-09-16T06:07:13.633 INFO:tasks.workunit.client.0.smithi178.stdout: "has_msr_rules": 0 2024-09-16T06:07:13.633 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.633 INFO:tasks.workunit.client.0.smithi178.stdout: "choose_args": {} 2024-09-16T06:07:13.633 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.633 INFO:tasks.workunit.client.0.smithi178.stdout: "fsmap": { 2024-09-16T06:07:13.633 INFO:tasks.workunit.client.0.smithi178.stdout: "epoch": 1, 2024-09-16T06:07:13.633 INFO:tasks.workunit.client.0.smithi178.stdout: "btime": "2024-09-16T06:06:28:080096+0000", 2024-09-16T06:07:13.633 INFO:tasks.workunit.client.0.smithi178.stdout: "default_fscid": -1, 2024-09-16T06:07:13.634 INFO:tasks.workunit.client.0.smithi178.stdout: "compat": { 2024-09-16T06:07:13.634 INFO:tasks.workunit.client.0.smithi178.stdout: "compat": {}, 2024-09-16T06:07:13.634 INFO:tasks.workunit.client.0.smithi178.stdout: "ro_compat": {}, 2024-09-16T06:07:13.634 INFO:tasks.workunit.client.0.smithi178.stdout: "incompat": { 2024-09-16T06:07:13.634 INFO:tasks.workunit.client.0.smithi178.stdout: "feature_1": "base v0.20", 2024-09-16T06:07:13.634 INFO:tasks.workunit.client.0.smithi178.stdout: "feature_2": "client writeable ranges", 2024-09-16T06:07:13.634 INFO:tasks.workunit.client.0.smithi178.stdout: "feature_3": "default file layouts on dirs", 2024-09-16T06:07:13.634 INFO:tasks.workunit.client.0.smithi178.stdout: "feature_4": "dir inode in separate object", 2024-09-16T06:07:13.634 INFO:tasks.workunit.client.0.smithi178.stdout: "feature_5": "mds uses versioned encoding", 2024-09-16T06:07:13.634 INFO:tasks.workunit.client.0.smithi178.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-16T06:07:13.634 INFO:tasks.workunit.client.0.smithi178.stdout: "feature_8": "no anchor table", 2024-09-16T06:07:13.635 INFO:tasks.workunit.client.0.smithi178.stdout: "feature_9": "file layout v2", 2024-09-16T06:07:13.635 INFO:tasks.workunit.client.0.smithi178.stdout: "feature_10": "snaprealm v2", 2024-09-16T06:07:13.635 INFO:tasks.workunit.client.0.smithi178.stdout: "feature_11": "minor log segments", 2024-09-16T06:07:13.635 INFO:tasks.workunit.client.0.smithi178.stdout: "feature_12": "quiesce subvolumes" 2024-09-16T06:07:13.635 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:07:13.635 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.635 INFO:tasks.workunit.client.0.smithi178.stdout: "feature_flags": { 2024-09-16T06:07:13.635 INFO:tasks.workunit.client.0.smithi178.stdout: "enable_multiple": true, 2024-09-16T06:07:13.635 INFO:tasks.workunit.client.0.smithi178.stdout: "ever_enabled_multiple": true 2024-09-16T06:07:13.635 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.635 INFO:tasks.workunit.client.0.smithi178.stdout: "standbys": [], 2024-09-16T06:07:13.635 INFO:tasks.workunit.client.0.smithi178.stdout: "filesystems": [] 2024-09-16T06:07:13.635 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.635 INFO:tasks.workunit.client.0.smithi178.stdout: "mdsmap_first_committed": 1, 2024-09-16T06:07:13.636 INFO:tasks.workunit.client.0.smithi178.stdout: "mdsmap_last_committed": 1, 2024-09-16T06:07:13.636 INFO:tasks.workunit.client.0.smithi178.stdout: "auth": { 2024-09-16T06:07:13.636 INFO:tasks.workunit.client.0.smithi178.stdout: "first_committed": 1, 2024-09-16T06:07:13.636 INFO:tasks.workunit.client.0.smithi178.stdout: "last_committed": 3, 2024-09-16T06:07:13.636 INFO:tasks.workunit.client.0.smithi178.stdout: "num_secrets": 1 2024-09-16T06:07:13.636 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.636 INFO:tasks.workunit.client.0.smithi178.stdout: "num_pg": 4, 2024-09-16T06:07:13.636 INFO:tasks.workunit.client.0.smithi178.stdout: "num_pg_active": 4, 2024-09-16T06:07:13.636 INFO:tasks.workunit.client.0.smithi178.stdout: "num_pg_unknown": 0, 2024-09-16T06:07:13.636 INFO:tasks.workunit.client.0.smithi178.stdout: "num_osd": 1, 2024-09-16T06:07:13.636 INFO:tasks.workunit.client.0.smithi178.stdout: "pool_sum": { 2024-09-16T06:07:13.636 INFO:tasks.workunit.client.0.smithi178.stdout: "stat_sum": { 2024-09-16T06:07:13.636 INFO:tasks.workunit.client.0.smithi178.stdout: "num_bytes": 19, 2024-09-16T06:07:13.636 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects": 2, 2024-09-16T06:07:13.637 INFO:tasks.workunit.client.0.smithi178.stdout: "num_object_clones": 0, 2024-09-16T06:07:13.637 INFO:tasks.workunit.client.0.smithi178.stdout: "num_object_copies": 4, 2024-09-16T06:07:13.637 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_missing_on_primary": 0, 2024-09-16T06:07:13.637 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_missing": 0, 2024-09-16T06:07:13.637 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_degraded": 2, 2024-09-16T06:07:13.637 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_misplaced": 0, 2024-09-16T06:07:13.637 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_unfound": 0, 2024-09-16T06:07:13.637 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_dirty": 2, 2024-09-16T06:07:13.637 INFO:tasks.workunit.client.0.smithi178.stdout: "num_whiteouts": 0, 2024-09-16T06:07:13.637 INFO:tasks.workunit.client.0.smithi178.stdout: "num_read": 0, 2024-09-16T06:07:13.637 INFO:tasks.workunit.client.0.smithi178.stdout: "num_read_kb": 0, 2024-09-16T06:07:13.637 INFO:tasks.workunit.client.0.smithi178.stdout: "num_write": 2, 2024-09-16T06:07:13.637 INFO:tasks.workunit.client.0.smithi178.stdout: "num_write_kb": 2, 2024-09-16T06:07:13.638 INFO:tasks.workunit.client.0.smithi178.stdout: "num_scrub_errors": 0, 2024-09-16T06:07:13.638 INFO:tasks.workunit.client.0.smithi178.stdout: "num_shallow_scrub_errors": 0, 2024-09-16T06:07:13.638 INFO:tasks.workunit.client.0.smithi178.stdout: "num_deep_scrub_errors": 0, 2024-09-16T06:07:13.638 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_recovered": 0, 2024-09-16T06:07:13.638 INFO:tasks.workunit.client.0.smithi178.stdout: "num_bytes_recovered": 0, 2024-09-16T06:07:13.638 INFO:tasks.workunit.client.0.smithi178.stdout: "num_keys_recovered": 0, 2024-09-16T06:07:13.638 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_omap": 0, 2024-09-16T06:07:13.638 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_hit_set_archive": 0, 2024-09-16T06:07:13.638 INFO:tasks.workunit.client.0.smithi178.stdout: "num_bytes_hit_set_archive": 0, 2024-09-16T06:07:13.638 INFO:tasks.workunit.client.0.smithi178.stdout: "num_flush": 0, 2024-09-16T06:07:13.638 INFO:tasks.workunit.client.0.smithi178.stdout: "num_flush_kb": 0, 2024-09-16T06:07:13.638 INFO:tasks.workunit.client.0.smithi178.stdout: "num_evict": 0, 2024-09-16T06:07:13.638 INFO:tasks.workunit.client.0.smithi178.stdout: "num_evict_kb": 0, 2024-09-16T06:07:13.638 INFO:tasks.workunit.client.0.smithi178.stdout: "num_promote": 0, 2024-09-16T06:07:13.638 INFO:tasks.workunit.client.0.smithi178.stdout: "num_flush_mode_high": 0, 2024-09-16T06:07:13.639 INFO:tasks.workunit.client.0.smithi178.stdout: "num_flush_mode_low": 0, 2024-09-16T06:07:13.639 INFO:tasks.workunit.client.0.smithi178.stdout: "num_evict_mode_some": 0, 2024-09-16T06:07:13.639 INFO:tasks.workunit.client.0.smithi178.stdout: "num_evict_mode_full": 0, 2024-09-16T06:07:13.639 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_pinned": 0, 2024-09-16T06:07:13.639 INFO:tasks.workunit.client.0.smithi178.stdout: "num_legacy_snapsets": 0, 2024-09-16T06:07:13.639 INFO:tasks.workunit.client.0.smithi178.stdout: "num_large_omap_objects": 0, 2024-09-16T06:07:13.639 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_manifest": 0, 2024-09-16T06:07:13.639 INFO:tasks.workunit.client.0.smithi178.stdout: "num_omap_bytes": 0, 2024-09-16T06:07:13.639 INFO:tasks.workunit.client.0.smithi178.stdout: "num_omap_keys": 0, 2024-09-16T06:07:13.639 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_repaired": 0 2024-09-16T06:07:13.639 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.639 INFO:tasks.workunit.client.0.smithi178.stdout: "store_stats": { 2024-09-16T06:07:13.639 INFO:tasks.workunit.client.0.smithi178.stdout: "total": 0, 2024-09-16T06:07:13.639 INFO:tasks.workunit.client.0.smithi178.stdout: "available": 0, 2024-09-16T06:07:13.640 INFO:tasks.workunit.client.0.smithi178.stdout: "internally_reserved": 0, 2024-09-16T06:07:13.640 INFO:tasks.workunit.client.0.smithi178.stdout: "allocated": 0, 2024-09-16T06:07:13.640 INFO:tasks.workunit.client.0.smithi178.stdout: "data_stored": 0, 2024-09-16T06:07:13.640 INFO:tasks.workunit.client.0.smithi178.stdout: "data_compressed": 0, 2024-09-16T06:07:13.640 INFO:tasks.workunit.client.0.smithi178.stdout: "data_compressed_allocated": 0, 2024-09-16T06:07:13.640 INFO:tasks.workunit.client.0.smithi178.stdout: "data_compressed_original": 0, 2024-09-16T06:07:13.640 INFO:tasks.workunit.client.0.smithi178.stdout: "omap_allocated": 0, 2024-09-16T06:07:13.640 INFO:tasks.workunit.client.0.smithi178.stdout: "internal_metadata": 0 2024-09-16T06:07:13.640 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.640 INFO:tasks.workunit.client.0.smithi178.stdout: "log_size": 3, 2024-09-16T06:07:13.640 INFO:tasks.workunit.client.0.smithi178.stdout: "ondisk_log_size": 3, 2024-09-16T06:07:13.640 INFO:tasks.workunit.client.0.smithi178.stdout: "up": 4, 2024-09-16T06:07:13.640 INFO:tasks.workunit.client.0.smithi178.stdout: "acting": 4, 2024-09-16T06:07:13.640 INFO:tasks.workunit.client.0.smithi178.stdout: "num_store_stats": 0 2024-09-16T06:07:13.641 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.641 INFO:tasks.workunit.client.0.smithi178.stdout: "osd_sum": { 2024-09-16T06:07:13.641 INFO:tasks.workunit.client.0.smithi178.stdout: "up_from": 0, 2024-09-16T06:07:13.641 INFO:tasks.workunit.client.0.smithi178.stdout: "seq": 0, 2024-09-16T06:07:13.641 INFO:tasks.workunit.client.0.smithi178.stdout: "num_pgs": 4, 2024-09-16T06:07:13.641 INFO:tasks.workunit.client.0.smithi178.stdout: "num_osds": 1, 2024-09-16T06:07:13.641 INFO:tasks.workunit.client.0.smithi178.stdout: "num_per_pool_osds": 1, 2024-09-16T06:07:13.641 INFO:tasks.workunit.client.0.smithi178.stdout: "num_per_pool_omap_osds": 1, 2024-09-16T06:07:13.641 INFO:tasks.workunit.client.0.smithi178.stdout: "kb": 104857600, 2024-09-16T06:07:13.641 INFO:tasks.workunit.client.0.smithi178.stdout: "kb_used": 436572, 2024-09-16T06:07:13.641 INFO:tasks.workunit.client.0.smithi178.stdout: "kb_used_data": 108, 2024-09-16T06:07:13.641 INFO:tasks.workunit.client.0.smithi178.stdout: "kb_used_omap": 1, 2024-09-16T06:07:13.641 INFO:tasks.workunit.client.0.smithi178.stdout: "kb_used_meta": 26814, 2024-09-16T06:07:13.641 INFO:tasks.workunit.client.0.smithi178.stdout: "kb_avail": 104421028, 2024-09-16T06:07:13.641 INFO:tasks.workunit.client.0.smithi178.stdout: "statfs": { 2024-09-16T06:07:13.642 INFO:tasks.workunit.client.0.smithi178.stdout: "total": 107374182400, 2024-09-16T06:07:13.642 INFO:tasks.workunit.client.0.smithi178.stdout: "available": 106927132672, 2024-09-16T06:07:13.642 INFO:tasks.workunit.client.0.smithi178.stdout: "internally_reserved": 0, 2024-09-16T06:07:13.642 INFO:tasks.workunit.client.0.smithi178.stdout: "allocated": 110592, 2024-09-16T06:07:13.642 INFO:tasks.workunit.client.0.smithi178.stdout: "data_stored": 22938, 2024-09-16T06:07:13.642 INFO:tasks.workunit.client.0.smithi178.stdout: "data_compressed": 0, 2024-09-16T06:07:13.642 INFO:tasks.workunit.client.0.smithi178.stdout: "data_compressed_allocated": 0, 2024-09-16T06:07:13.642 INFO:tasks.workunit.client.0.smithi178.stdout: "data_compressed_original": 0, 2024-09-16T06:07:13.642 INFO:tasks.workunit.client.0.smithi178.stdout: "omap_allocated": 1595, 2024-09-16T06:07:13.642 INFO:tasks.workunit.client.0.smithi178.stdout: "internal_metadata": 27457989 2024-09-16T06:07:13.642 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.642 INFO:tasks.workunit.client.0.smithi178.stdout: "hb_peers": [], 2024-09-16T06:07:13.642 INFO:tasks.workunit.client.0.smithi178.stdout: "snap_trim_queue_len": 0, 2024-09-16T06:07:13.642 INFO:tasks.workunit.client.0.smithi178.stdout: "num_snap_trimming": 0, 2024-09-16T06:07:13.642 INFO:tasks.workunit.client.0.smithi178.stdout: "num_shards_repaired": 0, 2024-09-16T06:07:13.643 INFO:tasks.workunit.client.0.smithi178.stdout: "op_queue_age_hist": { 2024-09-16T06:07:13.643 INFO:tasks.workunit.client.0.smithi178.stdout: "histogram": [], 2024-09-16T06:07:13.643 INFO:tasks.workunit.client.0.smithi178.stdout: "upper_bound": 1 2024-09-16T06:07:13.643 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.643 INFO:tasks.workunit.client.0.smithi178.stdout: "perf_stat": { 2024-09-16T06:07:13.643 INFO:tasks.workunit.client.0.smithi178.stdout: "commit_latency_ms": 0, 2024-09-16T06:07:13.643 INFO:tasks.workunit.client.0.smithi178.stdout: "apply_latency_ms": 0, 2024-09-16T06:07:13.643 INFO:tasks.workunit.client.0.smithi178.stdout: "commit_latency_ns": 0, 2024-09-16T06:07:13.643 INFO:tasks.workunit.client.0.smithi178.stdout: "apply_latency_ns": 0 2024-09-16T06:07:13.643 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.643 INFO:tasks.workunit.client.0.smithi178.stdout: "alerts": [], 2024-09-16T06:07:13.643 INFO:tasks.workunit.client.0.smithi178.stdout: "network_ping_times": [] 2024-09-16T06:07:13.643 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.643 INFO:tasks.workunit.client.0.smithi178.stdout: "osd_sum_by_class": { 2024-09-16T06:07:13.644 INFO:tasks.workunit.client.0.smithi178.stdout: "hdd": { 2024-09-16T06:07:13.644 INFO:tasks.workunit.client.0.smithi178.stdout: "up_from": 0, 2024-09-16T06:07:13.644 INFO:tasks.workunit.client.0.smithi178.stdout: "seq": 0, 2024-09-16T06:07:13.644 INFO:tasks.workunit.client.0.smithi178.stdout: "num_pgs": 4, 2024-09-16T06:07:13.644 INFO:tasks.workunit.client.0.smithi178.stdout: "num_osds": 1, 2024-09-16T06:07:13.644 INFO:tasks.workunit.client.0.smithi178.stdout: "num_per_pool_osds": 1, 2024-09-16T06:07:13.644 INFO:tasks.workunit.client.0.smithi178.stdout: "num_per_pool_omap_osds": 1, 2024-09-16T06:07:13.644 INFO:tasks.workunit.client.0.smithi178.stdout: "kb": 104857600, 2024-09-16T06:07:13.644 INFO:tasks.workunit.client.0.smithi178.stdout: "kb_used": 436572, 2024-09-16T06:07:13.644 INFO:tasks.workunit.client.0.smithi178.stdout: "kb_used_data": 108, 2024-09-16T06:07:13.644 INFO:tasks.workunit.client.0.smithi178.stdout: "kb_used_omap": 1, 2024-09-16T06:07:13.645 INFO:tasks.workunit.client.0.smithi178.stdout: "kb_used_meta": 26814, 2024-09-16T06:07:13.645 INFO:tasks.workunit.client.0.smithi178.stdout: "kb_avail": 104421028, 2024-09-16T06:07:13.645 INFO:tasks.workunit.client.0.smithi178.stdout: "statfs": { 2024-09-16T06:07:13.645 INFO:tasks.workunit.client.0.smithi178.stdout: "total": 107374182400, 2024-09-16T06:07:13.645 INFO:tasks.workunit.client.0.smithi178.stdout: "available": 106927132672, 2024-09-16T06:07:13.645 INFO:tasks.workunit.client.0.smithi178.stdout: "internally_reserved": 0, 2024-09-16T06:07:13.645 INFO:tasks.workunit.client.0.smithi178.stdout: "allocated": 110592, 2024-09-16T06:07:13.645 INFO:tasks.workunit.client.0.smithi178.stdout: "data_stored": 22938, 2024-09-16T06:07:13.645 INFO:tasks.workunit.client.0.smithi178.stdout: "data_compressed": 0, 2024-09-16T06:07:13.645 INFO:tasks.workunit.client.0.smithi178.stdout: "data_compressed_allocated": 0, 2024-09-16T06:07:13.645 INFO:tasks.workunit.client.0.smithi178.stdout: "data_compressed_original": 0, 2024-09-16T06:07:13.645 INFO:tasks.workunit.client.0.smithi178.stdout: "omap_allocated": 1595, 2024-09-16T06:07:13.645 INFO:tasks.workunit.client.0.smithi178.stdout: "internal_metadata": 27457989 2024-09-16T06:07:13.646 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.646 INFO:tasks.workunit.client.0.smithi178.stdout: "hb_peers": [], 2024-09-16T06:07:13.646 INFO:tasks.workunit.client.0.smithi178.stdout: "snap_trim_queue_len": 0, 2024-09-16T06:07:13.646 INFO:tasks.workunit.client.0.smithi178.stdout: "num_snap_trimming": 0, 2024-09-16T06:07:13.646 INFO:tasks.workunit.client.0.smithi178.stdout: "num_shards_repaired": 0, 2024-09-16T06:07:13.646 INFO:tasks.workunit.client.0.smithi178.stdout: "op_queue_age_hist": { 2024-09-16T06:07:13.646 INFO:tasks.workunit.client.0.smithi178.stdout: "histogram": [], 2024-09-16T06:07:13.646 INFO:tasks.workunit.client.0.smithi178.stdout: "upper_bound": 1 2024-09-16T06:07:13.646 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.646 INFO:tasks.workunit.client.0.smithi178.stdout: "perf_stat": { 2024-09-16T06:07:13.646 INFO:tasks.workunit.client.0.smithi178.stdout: "commit_latency_ms": 0, 2024-09-16T06:07:13.646 INFO:tasks.workunit.client.0.smithi178.stdout: "apply_latency_ms": 0, 2024-09-16T06:07:13.646 INFO:tasks.workunit.client.0.smithi178.stdout: "commit_latency_ns": 0, 2024-09-16T06:07:13.647 INFO:tasks.workunit.client.0.smithi178.stdout: "apply_latency_ns": 0 2024-09-16T06:07:13.647 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.647 INFO:tasks.workunit.client.0.smithi178.stdout: "alerts": [], 2024-09-16T06:07:13.647 INFO:tasks.workunit.client.0.smithi178.stdout: "network_ping_times": [] 2024-09-16T06:07:13.647 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:07:13.647 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.647 INFO:tasks.workunit.client.0.smithi178.stdout: "pool_stats": [ 2024-09-16T06:07:13.647 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.647 INFO:tasks.workunit.client.0.smithi178.stdout: "poolid": 1, 2024-09-16T06:07:13.647 INFO:tasks.workunit.client.0.smithi178.stdout: "num_pg": 4, 2024-09-16T06:07:13.647 INFO:tasks.workunit.client.0.smithi178.stdout: "stat_sum": { 2024-09-16T06:07:13.647 INFO:tasks.workunit.client.0.smithi178.stdout: "num_bytes": 19, 2024-09-16T06:07:13.647 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects": 2, 2024-09-16T06:07:13.647 INFO:tasks.workunit.client.0.smithi178.stdout: "num_object_clones": 0, 2024-09-16T06:07:13.647 INFO:tasks.workunit.client.0.smithi178.stdout: "num_object_copies": 4, 2024-09-16T06:07:13.648 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_missing_on_primary": 0, 2024-09-16T06:07:13.648 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_missing": 0, 2024-09-16T06:07:13.648 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_degraded": 2, 2024-09-16T06:07:13.648 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_misplaced": 0, 2024-09-16T06:07:13.648 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_unfound": 0, 2024-09-16T06:07:13.648 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_dirty": 2, 2024-09-16T06:07:13.648 INFO:tasks.workunit.client.0.smithi178.stdout: "num_whiteouts": 0, 2024-09-16T06:07:13.648 INFO:tasks.workunit.client.0.smithi178.stdout: "num_read": 0, 2024-09-16T06:07:13.648 INFO:tasks.workunit.client.0.smithi178.stdout: "num_read_kb": 0, 2024-09-16T06:07:13.648 INFO:tasks.workunit.client.0.smithi178.stdout: "num_write": 2, 2024-09-16T06:07:13.649 INFO:tasks.workunit.client.0.smithi178.stdout: "num_write_kb": 2, 2024-09-16T06:07:13.649 INFO:tasks.workunit.client.0.smithi178.stdout: "num_scrub_errors": 0, 2024-09-16T06:07:13.649 INFO:tasks.workunit.client.0.smithi178.stdout: "num_shallow_scrub_errors": 0, 2024-09-16T06:07:13.649 INFO:tasks.workunit.client.0.smithi178.stdout: "num_deep_scrub_errors": 0, 2024-09-16T06:07:13.650 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_recovered": 0, 2024-09-16T06:07:13.650 INFO:tasks.workunit.client.0.smithi178.stdout: "num_bytes_recovered": 0, 2024-09-16T06:07:13.650 INFO:tasks.workunit.client.0.smithi178.stdout: "num_keys_recovered": 0, 2024-09-16T06:07:13.650 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_omap": 0, 2024-09-16T06:07:13.650 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_hit_set_archive": 0, 2024-09-16T06:07:13.650 INFO:tasks.workunit.client.0.smithi178.stdout: "num_bytes_hit_set_archive": 0, 2024-09-16T06:07:13.650 INFO:tasks.workunit.client.0.smithi178.stdout: "num_flush": 0, 2024-09-16T06:07:13.650 INFO:tasks.workunit.client.0.smithi178.stdout: "num_flush_kb": 0, 2024-09-16T06:07:13.650 INFO:tasks.workunit.client.0.smithi178.stdout: "num_evict": 0, 2024-09-16T06:07:13.651 INFO:tasks.workunit.client.0.smithi178.stdout: "num_evict_kb": 0, 2024-09-16T06:07:13.651 INFO:tasks.workunit.client.0.smithi178.stdout: "num_promote": 0, 2024-09-16T06:07:13.651 INFO:tasks.workunit.client.0.smithi178.stdout: "num_flush_mode_high": 0, 2024-09-16T06:07:13.651 INFO:tasks.workunit.client.0.smithi178.stdout: "num_flush_mode_low": 0, 2024-09-16T06:07:13.651 INFO:tasks.workunit.client.0.smithi178.stdout: "num_evict_mode_some": 0, 2024-09-16T06:07:13.651 INFO:tasks.workunit.client.0.smithi178.stdout: "num_evict_mode_full": 0, 2024-09-16T06:07:13.651 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_pinned": 0, 2024-09-16T06:07:13.651 INFO:tasks.workunit.client.0.smithi178.stdout: "num_legacy_snapsets": 0, 2024-09-16T06:07:13.651 INFO:tasks.workunit.client.0.smithi178.stdout: "num_large_omap_objects": 0, 2024-09-16T06:07:13.651 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_manifest": 0, 2024-09-16T06:07:13.651 INFO:tasks.workunit.client.0.smithi178.stdout: "num_omap_bytes": 0, 2024-09-16T06:07:13.651 INFO:tasks.workunit.client.0.smithi178.stdout: "num_omap_keys": 0, 2024-09-16T06:07:13.651 INFO:tasks.workunit.client.0.smithi178.stdout: "num_objects_repaired": 0 2024-09-16T06:07:13.651 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.651 INFO:tasks.workunit.client.0.smithi178.stdout: "store_stats": { 2024-09-16T06:07:13.652 INFO:tasks.workunit.client.0.smithi178.stdout: "total": 0, 2024-09-16T06:07:13.652 INFO:tasks.workunit.client.0.smithi178.stdout: "available": 0, 2024-09-16T06:07:13.652 INFO:tasks.workunit.client.0.smithi178.stdout: "internally_reserved": 0, 2024-09-16T06:07:13.652 INFO:tasks.workunit.client.0.smithi178.stdout: "allocated": 4096, 2024-09-16T06:07:13.652 INFO:tasks.workunit.client.0.smithi178.stdout: "data_stored": 19, 2024-09-16T06:07:13.652 INFO:tasks.workunit.client.0.smithi178.stdout: "data_compressed": 0, 2024-09-16T06:07:13.652 INFO:tasks.workunit.client.0.smithi178.stdout: "data_compressed_allocated": 0, 2024-09-16T06:07:13.652 INFO:tasks.workunit.client.0.smithi178.stdout: "data_compressed_original": 0, 2024-09-16T06:07:13.652 INFO:tasks.workunit.client.0.smithi178.stdout: "omap_allocated": 0, 2024-09-16T06:07:13.652 INFO:tasks.workunit.client.0.smithi178.stdout: "internal_metadata": 0 2024-09-16T06:07:13.652 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.652 INFO:tasks.workunit.client.0.smithi178.stdout: "log_size": 3, 2024-09-16T06:07:13.652 INFO:tasks.workunit.client.0.smithi178.stdout: "ondisk_log_size": 3, 2024-09-16T06:07:13.653 INFO:tasks.workunit.client.0.smithi178.stdout: "up": 4, 2024-09-16T06:07:13.653 INFO:tasks.workunit.client.0.smithi178.stdout: "acting": 4, 2024-09-16T06:07:13.653 INFO:tasks.workunit.client.0.smithi178.stdout: "num_store_stats": 1 2024-09-16T06:07:13.653 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:07:13.653 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:07:13.653 INFO:tasks.workunit.client.0.smithi178.stdout: "osd_stats": [ 2024-09-16T06:07:13.653 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.653 INFO:tasks.workunit.client.0.smithi178.stdout: "osd": 0, 2024-09-16T06:07:13.653 INFO:tasks.workunit.client.0.smithi178.stdout: "seq": 21474836482 2024-09-16T06:07:13.653 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:07:13.653 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:07:13.653 INFO:tasks.workunit.client.0.smithi178.stdout: "num_pg_by_state": [ 2024-09-16T06:07:13.653 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.653 INFO:tasks.workunit.client.0.smithi178.stdout: "state": "active+undersized+degraded", 2024-09-16T06:07:13.653 INFO:tasks.workunit.client.0.smithi178.stdout: "num": 2 2024-09-16T06:07:13.654 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.654 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.654 INFO:tasks.workunit.client.0.smithi178.stdout: "state": "active+undersized", 2024-09-16T06:07:13.654 INFO:tasks.workunit.client.0.smithi178.stdout: "num": 2 2024-09-16T06:07:13.654 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:07:13.654 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:07:13.654 INFO:tasks.workunit.client.0.smithi178.stdout: "num_pg_by_osd": [ 2024-09-16T06:07:13.654 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.654 INFO:tasks.workunit.client.0.smithi178.stdout: "osd": 0, 2024-09-16T06:07:13.654 INFO:tasks.workunit.client.0.smithi178.stdout: "num_primary_pg": 4, 2024-09-16T06:07:13.654 INFO:tasks.workunit.client.0.smithi178.stdout: "num_acting_pg": 4, 2024-09-16T06:07:13.654 INFO:tasks.workunit.client.0.smithi178.stdout: "num_up_not_acting_pg": 0 2024-09-16T06:07:13.654 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:07:13.654 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:07:13.654 INFO:tasks.workunit.client.0.smithi178.stdout: "purged_snaps": [ 2024-09-16T06:07:13.655 INFO:tasks.workunit.client.0.smithi178.stdout: { 2024-09-16T06:07:13.655 INFO:tasks.workunit.client.0.smithi178.stdout: "pool": 1, 2024-09-16T06:07:13.655 INFO:tasks.workunit.client.0.smithi178.stdout: "purged_snaps": {} 2024-09-16T06:07:13.655 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:07:13.655 INFO:tasks.workunit.client.0.smithi178.stdout: ], 2024-09-16T06:07:13.655 INFO:tasks.workunit.client.0.smithi178.stdout: "servicemap": { 2024-09-16T06:07:13.655 INFO:tasks.workunit.client.0.smithi178.stdout: "epoch": 1, 2024-09-16T06:07:13.655 INFO:tasks.workunit.client.0.smithi178.stdout: "modified": "2024-09-16T06:06:28.105062+0000", 2024-09-16T06:07:13.655 INFO:tasks.workunit.client.0.smithi178.stdout: "services": {} 2024-09-16T06:07:13.655 INFO:tasks.workunit.client.0.smithi178.stdout: }, 2024-09-16T06:07:13.655 INFO:tasks.workunit.client.0.smithi178.stdout: "mgrstat_first_committed": 1, 2024-09-16T06:07:13.655 INFO:tasks.workunit.client.0.smithi178.stdout: "mgrstat_last_committed": 6, 2024-09-16T06:07:13.655 INFO:tasks.workunit.client.0.smithi178.stdout: "logm_first_committed": 1, 2024-09-16T06:07:13.655 INFO:tasks.workunit.client.0.smithi178.stdout: "logm_last_committed": 77, 2024-09-16T06:07:13.656 INFO:tasks.workunit.client.0.smithi178.stdout: "paxos": { 2024-09-16T06:07:13.656 INFO:tasks.workunit.client.0.smithi178.stdout: "first_committed": 1, 2024-09-16T06:07:13.656 INFO:tasks.workunit.client.0.smithi178.stdout: "last_committed": 102, 2024-09-16T06:07:13.656 INFO:tasks.workunit.client.0.smithi178.stdout: "last_pn": 0, 2024-09-16T06:07:13.656 INFO:tasks.workunit.client.0.smithi178.stdout: "accepted_pn": 0 2024-09-16T06:07:13.656 INFO:tasks.workunit.client.0.smithi178.stdout: } 2024-09-16T06:07:13.656 INFO:tasks.workunit.client.0.smithi178.stdout:} 2024-09-16T06:07:13.660 INFO:tasks.workunit.client.0.smithi178.stderr:report 2441005424 2024-09-16T06:07:13.660 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-16T06:07:13.660 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1742: test_wait_for_peered: run_osd td/ceph-helpers 1 2024-09-16T06:07:13.660 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T06:07:13.660 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T06:07:13.660 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T06:07:13.660 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T06:07:13.660 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-09-16T06:07:13.660 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T06:07:13.660 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:07:13.660 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:07:13.660 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:07:13.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-16T06:07:13.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-16T06:07:13.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T06:07:13.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T06:07:13.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T06:07:13.661 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T06:07:13.661 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:07:13.661 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:07:13.661 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:07:13.661 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:07:13.661 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:07:13.661 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:07:13.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:07:13.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:07:13.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T06:07:13.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T06:07:13.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T06:07:13.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T06:07:13.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:07:13.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:07:13.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:07:13.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:07:13.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T06:07:13.663 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T06:07:13.663 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-09-16T06:07:13.663 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T06:07:13.663 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=253005cf-e0ed-46ae-953a-91b1a68f9587 2024-09-16T06:07:13.663 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 253005cf-e0ed-46ae-953a-91b1a68f9587' 2024-09-16T06:07:13.663 INFO:tasks.workunit.client.0.smithi178.stdout:add osd1 253005cf-e0ed-46ae-953a-91b1a68f9587 2024-09-16T06:07:13.664 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T06:07:13.664 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQARy+dmP/3LJBAAPyV7XA8qagl7GgOZX1z2Tw== 2024-09-16T06:07:13.664 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQARy+dmP/3LJBAAPyV7XA8qagl7GgOZX1z2Tw=="}' 2024-09-16T06:07:13.664 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 253005cf-e0ed-46ae-953a-91b1a68f9587 -i td/ceph-helpers/1/new.json 2024-09-16T06:07:14.087 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:07:14.098 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-09-16T06:07:14.099 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQARy+dmP/3LJBAAPyV7XA8qagl7GgOZX1z2Tw== --osd-uuid 253005cf-e0ed-46ae-953a-91b1a68f9587 2024-09-16T06:07:14.129 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:07:14.128+0000 7f10a0883640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:07:14.141 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:07:14.140+0000 7f10a0883640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:07:14.141 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:07:14.141+0000 7f10a0883640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:07:14.142 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:07:14.141+0000 7f10a0883640 -1 bdev(0x55c828f83c00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2024-09-16T06:07:14.142 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:07:14.141+0000 7f10a0883640 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-16T06:07:19.962 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-16T06:07:19.962 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T06:07:19.963 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd1 key to auth repository 2024-09-16T06:07:19.963 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T06:07:19.964 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T06:07:20.487 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.1 2024-09-16T06:07:20.487 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T06:07:20.488 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T06:07:20.488 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T06:07:20.488 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T06:07:20.488 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T06:07:20.515 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:07:20.514+0000 7f075dc14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:07:20.535 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:07:20.534+0000 7f075dc14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:07:20.536 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:07:20.535+0000 7f075dc14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:07:20.924 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T06:07:20.924 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:07:20.924 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T06:07:20.924 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:07:20.925 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:07:20.925 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:07:20.925 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:07:20.925 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:07:20.925 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:07:20.926 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:07:21.296 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:07:21.877 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:07:21.877+0000 7f075dc14640 -1 Falling back to public interface 2024-09-16T06:07:22.297 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:07:22.297 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:07:22.298 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T06:07:22.298 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:07:22.298 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:07:22.298 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:07:22.709 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:07:23.710 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:07:23.710 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:07:23.710 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:07:23.711 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T06:07:23.711 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:07:23.711 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:07:24.095 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:07:24.505 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:07:24.504+0000 7f075dc14640 -1 osd.1 0 log_to_monitors true 2024-09-16T06:07:25.097 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:07:25.097 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:07:25.097 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T06:07:25.097 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T06:07:25.098 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:07:25.098 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:07:25.563 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:07:26.565 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:07:26.565 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:07:26.565 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T06:07:26.566 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T06:07:26.566 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:07:26.566 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:07:27.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:07:28.032 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:07:28.032 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T06:07:28.032 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:07:28.032 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T06:07:28.033 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:07:28.033 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:07:28.511 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:07:29.513 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:07:29.513 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T06:07:29.514 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:07:29.514 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T06:07:29.514 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:07:29.514 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:07:29.900 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:07:30.901 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:07:30.901 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:07:30.901 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T06:07:30.902 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T06:07:30.902 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:07:30.902 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:07:31.384 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:07:32.385 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:07:32.385 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:07:32.385 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T06:07:32.386 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T06:07:32.386 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:07:32.386 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:07:32.906 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:07:33.907 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:07:33.908 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:07:33.908 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T06:07:33.908 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T06:07:33.908 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:07:33.909 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:07:34.337 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:07:35.338 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:07:35.339 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T06:07:35.339 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:07:35.339 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T06:07:35.339 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:07:35.339 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:07:35.744 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:07:36.745 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:07:36.746 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:07:36.746 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T06:07:36.746 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T06:07:36.746 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:07:36.747 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:07:37.548 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:07:38.549 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:07:38.549 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:07:38.550 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T06:07:38.550 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T06:07:38.550 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:07:38.550 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:07:39.743 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:07:40.744 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:07:40.745 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T06:07:40.745 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:07:40.745 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T06:07:40.745 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:07:40.745 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:07:41.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:07:42.625 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:07:42.625 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:07:42.625 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T06:07:42.626 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T06:07:42.626 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:07:42.626 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:07:43.292 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:07:44.293 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:07:44.294 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T06:07:44.294 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:07:44.294 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T06:07:44.294 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:07:44.294 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:07:45.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:07:46.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:07:46.032 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:07:46.032 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T06:07:46.032 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T06:07:46.032 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:07:46.032 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:07:46.746 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:07:47.747 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:07:47.748 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:07:47.748 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T06:07:47.748 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T06:07:47.748 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:07:47.748 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:07:48.224 INFO:tasks.workunit.client.0.smithi178.stdout:osd.1 up in weight 1 up_from 16 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/864286278,v1:127.0.0.1:6811/864286278] [v2:127.0.0.1:6812/864286278,v1:127.0.0.1:6813/864286278] exists,up 253005cf-e0ed-46ae-953a-91b1a68f9587 2024-09-16T06:07:48.226 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:07:48.226 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:07:48.226 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:07:48.226 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1743: test_wait_for_peered: wait_for_peered 2024-09-16T06:07:48.226 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: wait_for_peered: local cmd= 2024-09-16T06:07:48.226 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: wait_for_peered: local num_peered=-1 2024-09-16T06:07:48.226 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1701: wait_for_peered: local cur_peered 2024-09-16T06:07:48.226 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: get_timeout_delays 90 .1 2024-09-16T06:07:48.226 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T06:07:48.227 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T06:07:48.227 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T06:07:48.227 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T06:07:48.227 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T06:07:48.310 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T06:07:48.310 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: local -a delays 2024-09-16T06:07:48.310 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_peered: local -i loop=0 2024-09-16T06:07:48.311 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1705: wait_for_peered: flush_pg_stats 2024-09-16T06:07:48.311 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-16T06:07:48.311 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-16T06:07:48.878 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-16T06:07:48.878 INFO:tasks.workunit.client.0.smithi178.stderr:1' 2024-09-16T06:07:48.878 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-16T06:07:48.878 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T06:07:48.878 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T06:07:49.080 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836491 2024-09-16T06:07:49.080 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836491 2024-09-16T06:07:49.080 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836491' 2024-09-16T06:07:49.080 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T06:07:49.080 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T06:07:49.222 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=68719476741 2024-09-16T06:07:49.222 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 68719476741 2024-09-16T06:07:49.222 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836491 1-68719476741' 2024-09-16T06:07:49.222 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T06:07:49.223 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836491 2024-09-16T06:07:49.223 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T06:07:49.223 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-16T06:07:49.224 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836491 2024-09-16T06:07:49.224 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T06:07:49.224 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836491 2024-09-16T06:07:49.225 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.0 seq 21474836491 2024-09-16T06:07:49.225 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2024-09-16T06:07:49.225 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T06:07:49.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836489 -lt 21474836491 2024-09-16T06:07:49.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-16T06:07:50.633 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T06:07:50.633 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T06:07:51.050 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836491 2024-09-16T06:07:51.051 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T06:07:51.051 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-68719476741 2024-09-16T06:07:51.051 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T06:07:51.052 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-09-16T06:07:51.052 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-68719476741 2024-09-16T06:07:51.052 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T06:07:51.053 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=68719476741 2024-09-16T06:07:51.053 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 68719476741' 2024-09-16T06:07:51.054 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.1 seq 68719476741 2024-09-16T06:07:51.054 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T06:07:51.467 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 68719476741 -lt 68719476741 2024-09-16T06:07:51.468 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: get_num_pgs 2024-09-16T06:07:51.468 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T06:07:51.468 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T06:07:51.968 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: test 4 == 0 2024-09-16T06:07:51.968 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_peered: true 2024-09-16T06:07:51.968 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: get_num_active_or_peered 2024-09-16T06:07:51.968 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1394: get_num_active_or_peered: local expression 2024-09-16T06:07:51.968 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-09-16T06:07:51.968 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-09-16T06:07:51.968 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: get_num_active_or_peered: ceph --format json pg dump pgs 2024-09-16T06:07:51.968 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-09-16T06:07:52.366 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: cur_peered=4 2024-09-16T06:07:52.367 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: get_num_pgs 2024-09-16T06:07:52.367 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T06:07:52.367 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T06:07:52.867 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: test 4 = 4 2024-09-16T06:07:52.867 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: break 2024-09-16T06:07:52.867 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1730: wait_for_peered: return 0 2024-09-16T06:07:52.867 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1744: test_wait_for_peered: teardown td/ceph-helpers 2024-09-16T06:07:52.867 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T06:07:52.867 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T06:07:52.867 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T06:07:52.867 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:07:52.868 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:07:52.868 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:07:52.868 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:07:52.868 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:07:52.973 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:07:52.973 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:07:52.974 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:07:52.974 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:07:52.975 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:07:52.975 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:07:52.975 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:07:52.976 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:07:52.976 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:07:52.977 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:07:52.977 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:07:52.977 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:07:52.978 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T06:07:52.978 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T06:07:52.988 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:07:52.988 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:07:52.988 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:07:52.988 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T06:07:52.990 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:07:52.990 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:07:52.990 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T06:07:52.990 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_wait_for_scrub td/ceph-helpers 2024-09-16T06:07:52.991 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: test_wait_for_scrub: local dir=td/ceph-helpers 2024-09-16T06:07:52.991 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: test_wait_for_scrub: setup td/ceph-helpers 2024-09-16T06:07:52.991 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T06:07:52.991 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T06:07:52.991 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T06:07:52.991 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T06:07:52.991 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T06:07:52.991 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:07:52.991 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:07:52.991 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:07:52.991 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:07:52.991 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:07:52.992 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:07:52.992 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:07:52.992 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:07:52.992 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:07:52.994 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:07:52.994 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:07:52.994 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:07:52.995 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:07:52.995 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:07:52.995 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:07:52.995 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:07:52.996 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:07:52.997 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T06:07:52.997 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T06:07:52.998 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:07:52.998 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:07:52.998 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:07:52.998 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T06:07:52.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:07:52.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:07:52.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T06:07:53.000 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T06:07:53.000 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:07:53.000 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:07:53.001 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T06:07:53.002 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T06:07:53.002 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T06:07:53.002 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T06:07:53.002 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T06:07:53.003 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: test_wait_for_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T06:07:53.003 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T06:07:53.003 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T06:07:53.003 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T06:07:53.003 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T06:07:53.003 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T06:07:53.003 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T06:07:53.384 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T06:07:53.384 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:07:53.384 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:07:53.385 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:07:53.385 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:07:53.385 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:07:53.385 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:07:53.385 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T06:07:53.577 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T06:07:53.577 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T06:07:53.578 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T06:07:53.578 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T06:07:53.578 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T06:07:53.579 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T06:07:53.579 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T06:07:53.579 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T06:07:53.579 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T06:07:53.579 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:07:53.579 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:07:53.579 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:07:53.579 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T06:07:53.580 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T06:07:53.580 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T06:07:53.661 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T06:07:53.661 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T06:07:53.661 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T06:07:53.661 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T06:07:53.661 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T06:07:53.661 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T06:07:53.661 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T06:07:53.661 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T06:07:53.661 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:07:53.662 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:07:53.662 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:07:53.662 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T06:07:53.662 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T06:07:53.662 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T06:07:53.736 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: test_wait_for_scrub: run_mgr td/ceph-helpers x 2024-09-16T06:07:53.737 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T06:07:53.737 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T06:07:53.737 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T06:07:53.737 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T06:07:53.737 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T06:07:53.737 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T06:07:54.581 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T06:07:54.581 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:07:54.581 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:07:54.581 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:07:54.581 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:07:54.581 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:07:54.581 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:07:54.582 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T06:07:54.582 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T06:07:54.613 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: test_wait_for_scrub: run_osd td/ceph-helpers 0 2024-09-16T06:07:54.613 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T06:07:54.613 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T06:07:54.613 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T06:07:54.613 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T06:07:54.613 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T06:07:54.614 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T06:07:54.614 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:07:54.614 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:07:54.614 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:07:54.614 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T06:07:54.614 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T06:07:54.614 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T06:07:54.614 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T06:07:54.614 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T06:07:54.615 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T06:07:54.615 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:07:54.615 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:07:54.615 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:07:54.615 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:07:54.615 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:07:54.615 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:07:54.615 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:07:54.615 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:07:54.615 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T06:07:54.615 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T06:07:54.615 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T06:07:54.615 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T06:07:54.616 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:07:54.616 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:07:54.616 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:07:54.616 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:07:54.616 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T06:07:54.616 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T06:07:54.616 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T06:07:54.616 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T06:07:54.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7b643da3-a7bd-4592-b739-acefdca7a7b8 2024-09-16T06:07:54.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 7b643da3-a7bd-4592-b739-acefdca7a7b8' 2024-09-16T06:07:54.617 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 7b643da3-a7bd-4592-b739-acefdca7a7b8 2024-09-16T06:07:54.618 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T06:07:54.633 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA6y+dmur6tJRAAcJMViromoBdS4fQK6fqrUg== 2024-09-16T06:07:54.633 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA6y+dmur6tJRAAcJMViromoBdS4fQK6fqrUg=="}' 2024-09-16T06:07:54.633 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7b643da3-a7bd-4592-b739-acefdca7a7b8 -i td/ceph-helpers/0/new.json 2024-09-16T06:07:54.935 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:07:54.945 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T06:07:54.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA6y+dmur6tJRAAcJMViromoBdS4fQK6fqrUg== --osd-uuid 7b643da3-a7bd-4592-b739-acefdca7a7b8 2024-09-16T06:07:54.974 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:07:54.973+0000 7f825e7fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:07:55.004 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:07:55.002+0000 7f825e7fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:07:55.005 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:07:55.004+0000 7f825e7fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:07:55.005 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:07:55.004+0000 7f825e7fd640 -1 bdev(0x55e664d89c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-16T06:07:55.005 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:07:55.004+0000 7f825e7fd640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T06:08:00.768 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T06:08:00.768 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T06:08:00.768 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T06:08:00.769 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T06:08:00.769 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T06:08:01.280 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T06:08:01.281 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T06:08:01.281 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T06:08:01.281 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T06:08:01.281 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T06:08:01.281 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T06:08:01.307 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:08:01.306+0000 7f56b7fae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:08:01.334 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:08:01.334+0000 7f56b7fae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:08:01.335 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:08:01.335+0000 7f56b7fae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:08:01.653 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T06:08:01.653 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:08:01.653 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T06:08:01.653 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:08:01.653 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:08:01.654 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:08:01.654 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:08:01.654 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:08:01.654 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:08:01.654 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:08:02.029 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:08:02.941 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:08:02.940+0000 7f56b7fae640 -1 Falling back to public interface 2024-09-16T06:08:03.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:08:03.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:08:03.032 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:08:03.032 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T06:08:03.033 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:08:03.033 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:08:03.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:08:04.442 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:08:04.442 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:08:04.442 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:08:04.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T06:08:04.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:08:04.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:08:04.868 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:08:05.596 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:08:05.595+0000 7f56b7fae640 -1 osd.0 0 log_to_monitors true 2024-09-16T06:08:05.870 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:08:05.870 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:08:05.870 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T06:08:05.871 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T06:08:05.871 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:08:05.871 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:08:06.334 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:08:07.336 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:08:07.336 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:08:07.336 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T06:08:07.337 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T06:08:07.338 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:08:07.338 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:08:07.752 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:08:08.754 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:08:08.754 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T06:08:08.755 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:08:08.755 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T06:08:08.755 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:08:08.755 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:08:09.154 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:08:10.155 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:08:10.156 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:08:10.156 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T06:08:10.157 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T06:08:10.157 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:08:10.157 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:08:10.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:08:11.537 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:08:11.537 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:08:11.538 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T06:08:11.538 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T06:08:11.538 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:08:11.538 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:08:11.917 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:08:12.918 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:08:12.919 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:08:12.919 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T06:08:12.919 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T06:08:12.919 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:08:12.920 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:08:13.336 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:08:14.337 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:08:14.337 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:08:14.337 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T06:08:14.338 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T06:08:14.338 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:08:14.338 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:08:14.751 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:08:15.753 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:08:15.753 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:08:15.753 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T06:08:15.754 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T06:08:15.754 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:08:15.754 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:08:16.143 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:08:17.145 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:08:17.145 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:08:17.145 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T06:08:17.146 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T06:08:17.146 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:08:17.146 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:08:17.902 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:08:18.903 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:08:18.903 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:08:18.904 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T06:08:18.904 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T06:08:18.904 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:08:18.904 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:08:19.704 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:08:20.705 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:08:20.706 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:08:20.706 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T06:08:20.707 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T06:08:20.707 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:08:20.707 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:08:21.850 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:08:22.851 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:08:22.852 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:08:22.852 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T06:08:22.852 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T06:08:22.852 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:08:22.853 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:08:23.715 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:08:24.716 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:08:24.716 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:08:24.716 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T06:08:24.717 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T06:08:24.717 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:08:24.717 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:08:25.249 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:08:26.250 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:08:26.250 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:08:26.251 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T06:08:26.251 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T06:08:26.251 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:08:26.252 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:08:27.219 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:08:28.221 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:08:28.221 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:08:28.221 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T06:08:28.222 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T06:08:28.222 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:08:28.222 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:08:28.627 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2995949296,v1:127.0.0.1:6803/2995949296] [v2:127.0.0.1:6804/2995949296,v1:127.0.0.1:6805/2995949296] exists,up 7b643da3-a7bd-4592-b739-acefdca7a7b8 2024-09-16T06:08:28.628 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:08:28.628 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:08:28.628 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:08:28.628 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: test_wait_for_scrub: create_rbd_pool 2024-09-16T06:08:28.628 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T06:08:29.019 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' does not exist 2024-09-16T06:08:29.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T06:08:29.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T06:08:29.730 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' already exists 2024-09-16T06:08:29.740 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T06:08:30.744 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T06:08:31.742 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: test_wait_for_scrub: wait_for_clean 2024-09-16T06:08:31.742 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T06:08:31.742 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T06:08:31.742 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T06:08:31.742 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T06:08:31.743 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T06:08:31.743 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T06:08:31.743 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T06:08:31.743 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T06:08:31.743 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T06:08:31.840 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T06:08:31.840 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T06:08:31.840 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T06:08:31.840 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T06:08:31.841 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-16T06:08:31.841 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-16T06:08:32.243 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-09-16T06:08:32.244 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-16T06:08:32.244 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T06:08:32.244 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T06:08:32.358 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836487 2024-09-16T06:08:32.358 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836487 2024-09-16T06:08:32.358 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487' 2024-09-16T06:08:32.358 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T06:08:32.359 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836487 2024-09-16T06:08:32.359 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T06:08:32.359 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-16T06:08:32.360 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836487 2024-09-16T06:08:32.360 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T06:08:32.361 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836487 2024-09-16T06:08:32.361 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-09-16T06:08:32.361 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.0 seq 21474836487 2024-09-16T06:08:32.361 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T06:08:32.736 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836487 -lt 21474836487 2024-09-16T06:08:32.737 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T06:08:32.737 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T06:08:32.737 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T06:08:33.261 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T06:08:33.261 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T06:08:33.261 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T06:08:33.261 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T06:08:33.261 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T06:08:33.261 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T06:08:33.261 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T06:08:33.262 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T06:08:33.655 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T06:08:33.655 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T06:08:33.655 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T06:08:33.656 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T06:08:34.243 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T06:08:34.243 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T06:08:34.243 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T06:08:34.243 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: test_wait_for_scrub: local pgid=1.0 2024-09-16T06:08:34.243 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2032: test_wait_for_scrub: ceph pg repair 1.0 2024-09-16T06:08:34.621 INFO:tasks.workunit.client.0.smithi178.stderr:instructing pg 1.0 on osd.0 to repair 2024-09-16T06:08:34.633 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: test_wait_for_scrub: get_last_scrub_stamp 1.0 2024-09-16T06:08:34.633 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T06:08:34.633 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T06:08:34.634 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T06:08:34.634 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T06:08:35.005 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: test_wait_for_scrub: local last_scrub=2024-09-16T06:08:29.360740+0000 2024-09-16T06:08:35.005 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2034: test_wait_for_scrub: wait_for_scrub 1.0 2024-09-16T06:08:29.360740+0000 2024-09-16T06:08:35.006 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-09-16T06:08:35.006 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-09-16T06:08:29.360740+0000 2024-09-16T06:08:35.006 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-09-16T06:08:35.006 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-09-16T06:08:35.006 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-09-16T06:08:35.006 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T06:08:35.006 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T06:08:35.006 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T06:08:35.006 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T06:08:35.006 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T06:08:35.370 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-09-16T06:08:29.360740+0000 '>' 2024-09-16T06:08:29.360740+0000 2024-09-16T06:08:35.370 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-09-16T06:08:36.372 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-09-16T06:08:36.372 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-09-16T06:08:36.373 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T06:08:36.373 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T06:08:36.373 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T06:08:36.373 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T06:08:36.373 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T06:08:36.748 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-09-16T06:08:29.360740+0000 '>' 2024-09-16T06:08:29.360740+0000 2024-09-16T06:08:36.748 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-09-16T06:08:37.749 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-09-16T06:08:37.750 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-09-16T06:08:37.750 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T06:08:37.750 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T06:08:37.750 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T06:08:37.750 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T06:08:37.750 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T06:08:38.124 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-09-16T06:08:29.360740+0000 '>' 2024-09-16T06:08:29.360740+0000 2024-09-16T06:08:38.124 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-09-16T06:08:39.125 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-09-16T06:08:39.125 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-09-16T06:08:39.126 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T06:08:39.126 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T06:08:39.126 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T06:08:39.126 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T06:08:39.126 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T06:08:39.494 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-09-16T06:08:29.360740+0000 '>' 2024-09-16T06:08:29.360740+0000 2024-09-16T06:08:39.494 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-09-16T06:08:40.495 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-09-16T06:08:40.495 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-09-16T06:08:40.496 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T06:08:40.496 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T06:08:40.496 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T06:08:40.496 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T06:08:40.496 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T06:08:40.871 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-09-16T06:08:34.733865+0000 '>' 2024-09-16T06:08:29.360740+0000 2024-09-16T06:08:40.871 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-09-16T06:08:40.871 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2035: test_wait_for_scrub: kill_daemons td/ceph-helpers KILL osd 2024-09-16T06:08:40.871 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:08:40.871 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:08:40.871 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:08:40.871 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:08:40.871 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:08:40.976 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:08:40.977 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2036: test_wait_for_scrub: get_last_scrub_stamp 1.0 2024-09-16T06:08:40.977 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T06:08:40.977 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T06:08:40.977 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T06:08:40.977 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T06:08:41.341 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2036: test_wait_for_scrub: last_scrub=2024-09-16T06:08:34.733865+0000 2024-09-16T06:08:41.341 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: test_wait_for_scrub: TIMEOUT=1 2024-09-16T06:08:41.341 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: test_wait_for_scrub: wait_for_scrub 1.0 2024-09-16T06:08:34.733865+0000 2024-09-16T06:08:41.341 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-09-16T06:08:41.341 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-09-16T06:08:34.733865+0000 2024-09-16T06:08:41.341 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-09-16T06:08:41.341 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-09-16T06:08:41.341 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 1 )) 2024-09-16T06:08:41.342 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T06:08:41.342 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T06:08:41.342 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T06:08:41.342 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T06:08:41.342 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T06:08:41.705 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-09-16T06:08:34.733865+0000 '>' 2024-09-16T06:08:34.733865+0000 2024-09-16T06:08:41.705 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-09-16T06:08:42.707 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-09-16T06:08:42.707 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 1 )) 2024-09-16T06:08:42.707 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: return 1 2024-09-16T06:08:42.707 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2038: test_wait_for_scrub: teardown td/ceph-helpers 2024-09-16T06:08:42.707 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T06:08:42.707 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T06:08:42.707 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T06:08:42.707 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:08:42.707 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:08:42.707 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:08:42.707 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:08:42.708 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:08:42.813 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:08:42.813 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:08:42.814 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:08:42.814 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:08:42.815 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:08:42.816 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:08:42.816 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:08:42.817 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:08:42.817 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:08:42.817 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:08:42.817 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:08:42.818 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:08:42.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T06:08:42.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T06:08:42.830 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:08:42.830 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:08:42.830 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:08:42.830 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T06:08:42.830 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:08:42.831 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:08:42.831 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2339: run_tests: for func in $funcs 2024-09-16T06:08:42.831 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: run_tests: test_wait_osd_id_used_by_pgs td/ceph-helpers 2024-09-16T06:08:42.831 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1501: test_wait_osd_id_used_by_pgs: local dir=td/ceph-helpers 2024-09-16T06:08:42.831 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1503: test_wait_osd_id_used_by_pgs: setup td/ceph-helpers 2024-09-16T06:08:42.831 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T06:08:42.831 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T06:08:42.831 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T06:08:42.831 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T06:08:42.831 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T06:08:42.831 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:08:42.831 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:08:42.831 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:08:42.832 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:08:42.832 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:08:42.832 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:08:42.832 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:08:42.832 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:08:42.833 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:08:42.833 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:08:42.833 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:08:42.833 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:08:42.833 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:08:42.833 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:08:42.834 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:08:42.834 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:08:42.834 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:08:42.836 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T06:08:42.836 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T06:08:42.837 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:08:42.837 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:08:42.837 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:08:42.837 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T06:08:42.838 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:08:42.838 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:08:42.838 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T06:08:42.840 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T06:08:42.840 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:08:42.840 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:08:42.840 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.77187 2024-09-16T06:08:42.842 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T06:08:42.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T06:08:42.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T06:08:42.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T06:08:42.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1504: test_wait_osd_id_used_by_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T06:08:42.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T06:08:42.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T06:08:42.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T06:08:42.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T06:08:42.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T06:08:42.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T06:08:43.233 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T06:08:43.233 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:08:43.233 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:08:43.233 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:08:43.233 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:08:43.233 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:08:43.233 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:08:43.234 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T06:08:43.434 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T06:08:43.435 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T06:08:43.435 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T06:08:43.435 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T06:08:43.435 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T06:08:43.435 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T06:08:43.435 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T06:08:43.436 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T06:08:43.436 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T06:08:43.436 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:08:43.437 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:08:43.437 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:08:43.437 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T06:08:43.437 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T06:08:43.437 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get fsid 2024-09-16T06:08:43.517 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T06:08:43.517 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T06:08:43.517 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T06:08:43.517 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T06:08:43.517 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T06:08:43.517 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T06:08:43.517 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T06:08:43.517 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T06:08:43.517 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:08:43.517 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:08:43.518 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:08:43.518 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.77187/ceph-mon.a.asok 2024-09-16T06:08:43.518 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T06:08:43.518 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.77187/ceph-mon.a.asok config get mon_host 2024-09-16T06:08:43.596 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1505: test_wait_osd_id_used_by_pgs: run_mgr td/ceph-helpers x 2024-09-16T06:08:43.596 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T06:08:43.596 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T06:08:43.597 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T06:08:43.597 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T06:08:43.597 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T06:08:43.597 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T06:08:44.446 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T06:08:44.446 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:08:44.447 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:08:44.447 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:08:44.447 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:08:44.447 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:08:44.447 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:08:44.447 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T06:08:44.448 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T06:08:44.480 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1506: test_wait_osd_id_used_by_pgs: run_osd td/ceph-helpers 0 2024-09-16T06:08:44.480 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T06:08:44.480 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T06:08:44.480 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T06:08:44.481 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T06:08:44.481 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T06:08:44.481 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T06:08:44.481 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:08:44.481 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:08:44.481 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:08:44.481 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T06:08:44.481 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T06:08:44.481 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T06:08:44.481 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T06:08:44.481 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T06:08:44.481 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T06:08:44.481 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:08:44.482 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:08:44.482 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:08:44.482 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:08:44.482 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:08:44.482 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:08:44.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' 2024-09-16T06:08:44.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:08:44.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T06:08:44.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T06:08:44.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T06:08:44.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T06:08:44.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:08:44.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:08:44.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:08:44.483 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:08:44.483 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T06:08:44.483 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T06:08:44.483 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T06:08:44.483 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T06:08:44.483 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=73481cb8-9749-484b-9147-dab195e74574 2024-09-16T06:08:44.483 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 73481cb8-9749-484b-9147-dab195e74574' 2024-09-16T06:08:44.484 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 73481cb8-9749-484b-9147-dab195e74574 2024-09-16T06:08:44.484 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T06:08:44.500 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBsy+dm3zfGHRAAMdNWqWWZJ+Svb1dshlacxw== 2024-09-16T06:08:44.500 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBsy+dm3zfGHRAAMdNWqWWZJ+Svb1dshlacxw=="}' 2024-09-16T06:08:44.500 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 73481cb8-9749-484b-9147-dab195e74574 -i td/ceph-helpers/0/new.json 2024-09-16T06:08:44.767 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:08:44.777 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T06:08:44.778 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBsy+dm3zfGHRAAMdNWqWWZJ+Svb1dshlacxw== --osd-uuid 73481cb8-9749-484b-9147-dab195e74574 2024-09-16T06:08:44.807 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:08:44.806+0000 7fb085416640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:08:44.853 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:08:44.852+0000 7fb085416640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:08:44.854 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:08:44.853+0000 7fb085416640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:08:44.854 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:08:44.854+0000 7fb085416640 -1 bdev(0x55817dfa1c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2024-09-16T06:08:44.854 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:08:44.854+0000 7fb085416640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T06:08:50.409 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T06:08:50.409 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T06:08:50.410 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T06:08:50.410 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T06:08:50.410 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T06:08:50.962 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T06:08:50.962 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T06:08:50.962 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=e36fe3b7-e285-4199-9297-eafea05d044b --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.77187/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T06:08:50.963 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T06:08:50.963 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T06:08:50.963 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T06:08:50.988 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:08:50.988+0000 7ffa33fcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:08:50.990 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:08:50.990+0000 7ffa33fcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:08:50.991 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:08:50.991+0000 7ffa33fcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:08:51.376 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T06:08:51.377 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:08:51.377 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T06:08:51.377 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:08:51.377 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:08:51.377 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:08:51.377 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:08:51.377 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:08:51.378 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:08:51.378 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:08:51.755 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:08:52.327 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:08:52.327+0000 7ffa33fcc640 -1 Falling back to public interface 2024-09-16T06:08:52.757 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:08:52.757 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:08:52.757 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:08:52.758 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T06:08:52.758 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:08:52.758 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:08:53.137 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:08:54.138 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:08:54.138 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:08:54.138 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:08:54.139 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T06:08:54.139 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:08:54.139 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:08:54.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:08:54.945 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:08:54.944+0000 7ffa33fcc640 -1 osd.0 0 log_to_monitors true 2024-09-16T06:08:55.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:08:55.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:08:55.530 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T06:08:55.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T06:08:55.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:08:55.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:08:56.023 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:08:57.024 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:08:57.024 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:08:57.025 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T06:08:57.025 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T06:08:57.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:08:57.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:08:57.406 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:08:58.408 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:08:58.408 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:08:58.408 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T06:08:58.408 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T06:08:58.409 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:08:58.409 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:08:58.844 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:08:59.845 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:08:59.845 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:08:59.845 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T06:08:59.846 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T06:08:59.846 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:08:59.846 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:09:00.246 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:09:01.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:09:01.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:09:01.248 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T06:09:01.249 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T06:09:01.249 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:09:01.249 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:09:01.722 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:09:02.723 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:09:02.723 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:09:02.724 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T06:09:02.724 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T06:09:02.724 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:09:02.725 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:09:03.104 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:09:04.106 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:09:04.106 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:09:04.106 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T06:09:04.107 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T06:09:04.107 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:09:04.107 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:09:04.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:09:05.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:09:05.531 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:09:05.531 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T06:09:05.531 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T06:09:05.531 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:09:05.531 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:09:06.027 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:09:07.029 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:09:07.029 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:09:07.029 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T06:09:07.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T06:09:07.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:09:07.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:09:07.605 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:09:08.606 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:09:08.607 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:09:08.607 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T06:09:08.607 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T06:09:08.608 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:09:08.608 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:09:09.551 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:09:10.552 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:09:10.552 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:09:10.553 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T06:09:10.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T06:09:10.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:09:10.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:09:11.238 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:09:12.239 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:09:12.240 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:09:12.240 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T06:09:12.240 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T06:09:12.240 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:09:12.241 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:09:12.743 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:09:13.744 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:09:13.745 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:09:13.745 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T06:09:13.745 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T06:09:13.746 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:09:13.746 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:09:14.428 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:09:15.434 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:09:15.434 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:09:15.434 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T06:09:15.435 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T06:09:15.435 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:09:15.435 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:09:16.222 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:09:17.223 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:09:17.223 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:09:17.224 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T06:09:17.224 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T06:09:17.224 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:09:17.224 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:09:17.671 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/326408395,v1:127.0.0.1:6803/326408395] [v2:127.0.0.1:6804/326408395,v1:127.0.0.1:6805/326408395] exists,up 73481cb8-9749-484b-9147-dab195e74574 2024-09-16T06:09:17.672 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:09:17.672 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:09:17.672 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:09:17.672 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1507: test_wait_osd_id_used_by_pgs: create_rbd_pool 2024-09-16T06:09:17.672 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T06:09:18.126 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' does not exist 2024-09-16T06:09:18.137 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T06:09:18.137 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T06:09:18.665 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'rbd' already exists 2024-09-16T06:09:18.676 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T06:09:19.677 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T06:09:20.929 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1508: test_wait_osd_id_used_by_pgs: wait_for_clean 2024-09-16T06:09:20.929 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T06:09:20.929 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T06:09:20.929 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T06:09:20.930 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T06:09:20.930 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T06:09:20.930 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T06:09:20.930 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T06:09:20.930 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T06:09:20.930 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T06:09:21.024 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T06:09:21.025 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T06:09:21.025 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T06:09:21.025 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T06:09:21.025 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-16T06:09:21.025 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-16T06:09:21.437 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-09-16T06:09:21.437 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-16T06:09:21.438 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T06:09:21.438 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T06:09:21.550 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836486 2024-09-16T06:09:21.551 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836486 2024-09-16T06:09:21.551 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836486' 2024-09-16T06:09:21.551 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T06:09:21.551 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836486 2024-09-16T06:09:21.551 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T06:09:21.552 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-16T06:09:21.552 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836486 2024-09-16T06:09:21.552 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T06:09:21.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836486 2024-09-16T06:09:21.554 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.0 seq 21474836486 2024-09-16T06:09:21.554 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-09-16T06:09:21.554 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T06:09:21.928 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836484 -lt 21474836486 2024-09-16T06:09:21.928 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-16T06:09:22.929 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T06:09:22.929 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T06:09:23.304 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836486 -lt 21474836486 2024-09-16T06:09:23.305 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T06:09:23.305 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T06:09:23.305 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T06:09:23.836 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T06:09:23.836 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T06:09:23.837 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T06:09:23.837 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T06:09:23.837 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T06:09:23.837 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T06:09:23.837 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T06:09:23.837 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T06:09:24.271 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T06:09:24.271 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T06:09:24.272 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T06:09:24.272 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T06:09:24.783 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T06:09:24.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T06:09:24.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T06:09:24.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1509: test_wait_osd_id_used_by_pgs: wait_osd_id_used_by_pgs 0 8 2024-09-16T06:09:24.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1484: wait_osd_id_used_by_pgs: local id=0 2024-09-16T06:09:24.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1485: wait_osd_id_used_by_pgs: local count=8 2024-09-16T06:09:24.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1487: wait_osd_id_used_by_pgs: status=1 2024-09-16T06:09:24.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1488: wait_osd_id_used_by_pgs: (( i=0 )) 2024-09-16T06:09:24.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1488: wait_osd_id_used_by_pgs: (( i < 300 / 5 )) 2024-09-16T06:09:24.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1489: wait_osd_id_used_by_pgs: echo 0 2024-09-16T06:09:24.785 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:09:24.785 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1490: wait_osd_id_used_by_pgs: get_osd_id_used_by_pgs 2024-09-16T06:09:24.785 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1490: wait_osd_id_used_by_pgs: grep -c 0 2024-09-16T06:09:24.785 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: ceph --format json pg dump pgs 2024-09-16T06:09:24.786 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: jq '.pg_stats | .[] | .up[], .acting[]' 2024-09-16T06:09:24.786 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: sort 2024-09-16T06:09:25.187 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1490: wait_osd_id_used_by_pgs: test 8 = 8 2024-09-16T06:09:25.187 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1493: wait_osd_id_used_by_pgs: status=0 2024-09-16T06:09:25.187 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: wait_osd_id_used_by_pgs: break 2024-09-16T06:09:25.187 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1497: wait_osd_id_used_by_pgs: return 0 2024-09-16T06:09:25.187 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1510: test_wait_osd_id_used_by_pgs: TIMEOUT=1 2024-09-16T06:09:25.187 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1510: test_wait_osd_id_used_by_pgs: wait_osd_id_used_by_pgs 123 5 2024-09-16T06:09:25.187 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1484: wait_osd_id_used_by_pgs: local id=123 2024-09-16T06:09:25.187 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1485: wait_osd_id_used_by_pgs: local count=5 2024-09-16T06:09:25.188 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1487: wait_osd_id_used_by_pgs: status=1 2024-09-16T06:09:25.188 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1488: wait_osd_id_used_by_pgs: (( i=0 )) 2024-09-16T06:09:25.188 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1488: wait_osd_id_used_by_pgs: (( i < 1 / 5 )) 2024-09-16T06:09:25.188 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1497: wait_osd_id_used_by_pgs: return 1 2024-09-16T06:09:25.188 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1511: test_wait_osd_id_used_by_pgs: teardown td/ceph-helpers 2024-09-16T06:09:25.188 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T06:09:25.188 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T06:09:25.188 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T06:09:25.188 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:09:25.188 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:09:25.188 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:09:25.188 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:09:25.188 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:09:25.292 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:09:25.292 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:09:25.293 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:09:25.293 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:09:25.294 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:09:25.294 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:09:25.294 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:09:25.295 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:09:25.295 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:09:25.295 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:09:25.296 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:09:25.296 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:09:25.297 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T06:09:25.297 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T06:09:25.303 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:09:25.304 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:09:25.304 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.77187 2024-09-16T06:09:25.304 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.77187 2024-09-16T06:09:25.305 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:09:25.305 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:09:25.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: : exit 0 2024-09-16T06:09:25.306 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T06:09:25.306 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T06:09:25.332 INFO:tasks.workunit:Running workunit misc/test-snaptrim-stats.sh... 2024-09-16T06:09:25.333 DEBUG:teuthology.orchestra.run.smithi178:workunit test misc/test-snaptrim-stats.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ad35db06e8924912f01e32b35576297c0f1352b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh 2024-09-16T06:09:25.394 INFO:tasks.workunit.client.0.smithi178.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-16T06:09:25.399 INFO:tasks.workunit.client.0.smithi178.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-16T06:09:25.399 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-16T06:09:25.399 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-16T06:09:25.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-09-16T06:09:25.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-09-16T06:09:25.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-09-16T06:09:25.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-09-16T06:09:25.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-09-16T06:09:25.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-09-16T06:09:25.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/test-snaptrim-stats 2024-09-16T06:09:25.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:21: run: local dir=td/test-snaptrim-stats 2024-09-16T06:09:25.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:22: run: shift 2024-09-16T06:09:25.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:24: run: export CEPH_MON=127.0.0.1:7124 2024-09-16T06:09:25.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:24: run: CEPH_MON=127.0.0.1:7124 2024-09-16T06:09:25.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:25: run: export CEPH_ARGS 2024-09-16T06:09:25.400 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:26: run: uuidgen 2024-09-16T06:09:25.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:26: run: CEPH_ARGS+='--fsid=9b40f4bd-1dcc-4b6e-878d-4e61738f3b4d --auth-supported=none ' 2024-09-16T06:09:25.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2024-09-16T06:09:25.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:28: run: CEPH_ARGS+='--debug-bluestore 20 ' 2024-09-16T06:09:25.403 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:30: run: set 2024-09-16T06:09:25.403 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-16T06:09:25.403 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:30: run: local 'funcs=TEST_snaptrim_stats 2024-09-16T06:09:25.403 INFO:tasks.workunit.client.0.smithi178.stderr:TEST_snaptrim_stats_multiple_snaps' 2024-09-16T06:09:25.403 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:31: run: for func in $funcs 2024-09-16T06:09:25.403 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:32: run: setup td/test-snaptrim-stats 2024-09-16T06:09:25.403 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-snaptrim-stats 2024-09-16T06:09:25.403 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-snaptrim-stats 2024-09-16T06:09:25.403 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-09-16T06:09:25.403 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T06:09:25.403 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-09-16T06:09:25.404 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:09:25.404 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:09:25.404 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:09:25.404 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:09:25.404 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:09:25.405 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:09:25.405 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:09:25.405 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:09:25.405 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:09:25.405 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:09:25.405 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:09:25.406 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:09:25.406 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:09:25.406 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:09:25.406 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:09:25.406 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:09:25.407 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:09:25.408 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T06:09:25.408 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-09-16T06:09:25.409 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:09:25.410 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:09:25.410 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176784 2024-09-16T06:09:25.410 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.176784 2024-09-16T06:09:25.410 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:09:25.410 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:09:25.410 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-snaptrim-stats 2024-09-16T06:09:25.411 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T06:09:25.411 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:09:25.412 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176784 2024-09-16T06:09:25.412 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.176784 2024-09-16T06:09:25.414 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T06:09:25.415 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-16T06:09:25.415 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-16T06:09:25.415 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T06:09:25.415 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-snaptrim-stats 1' TERM HUP INT 2024-09-16T06:09:25.415 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:33: run: TEST_snaptrim_stats td/test-snaptrim-stats 2024-09-16T06:09:25.415 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:39: TEST_snaptrim_stats: local dir=td/test-snaptrim-stats 2024-09-16T06:09:25.415 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:40: TEST_snaptrim_stats: local poolname=test 2024-09-16T06:09:25.415 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:41: TEST_snaptrim_stats: local OSDS=3 2024-09-16T06:09:25.415 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:42: TEST_snaptrim_stats: local PGNUM=8 2024-09-16T06:09:25.415 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:43: TEST_snaptrim_stats: local PGPNUM=8 2024-09-16T06:09:25.415 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:44: TEST_snaptrim_stats: local objects=10 2024-09-16T06:09:25.415 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:45: TEST_snaptrim_stats: local WAIT_FOR_UPDATE=10 2024-09-16T06:09:25.416 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:47: TEST_snaptrim_stats: setup td/test-snaptrim-stats 2024-09-16T06:09:25.416 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-snaptrim-stats 2024-09-16T06:09:25.416 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-snaptrim-stats 2024-09-16T06:09:25.416 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-09-16T06:09:25.416 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T06:09:25.416 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-09-16T06:09:25.416 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:09:25.416 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:09:25.416 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:09:25.416 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:09:25.416 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:09:25.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:09:25.417 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:09:25.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:09:25.417 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:09:25.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:09:25.417 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:09:25.418 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:09:25.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:09:25.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:09:25.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:09:25.418 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:09:25.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:09:25.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T06:09:25.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-09-16T06:09:25.421 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:09:25.421 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:09:25.422 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176784 2024-09-16T06:09:25.422 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.176784 2024-09-16T06:09:25.422 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:09:25.422 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:09:25.422 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-snaptrim-stats 2024-09-16T06:09:25.423 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T06:09:25.423 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:09:25.424 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176784 2024-09-16T06:09:25.424 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.176784 2024-09-16T06:09:25.425 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T06:09:25.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T06:09:25.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T06:09:25.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-snaptrim-stats 1' TERM HUP INT 2024-09-16T06:09:25.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:48: TEST_snaptrim_stats: run_mon td/test-snaptrim-stats a --osd_pool_default_size=3 2024-09-16T06:09:25.426 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-snaptrim-stats 2024-09-16T06:09:25.426 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T06:09:25.426 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T06:09:25.426 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T06:09:25.426 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-snaptrim-stats/a 2024-09-16T06:09:25.426 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-snaptrim-stats/a --run-dir=td/test-snaptrim-stats --osd_pool_default_size=3 2024-09-16T06:09:25.792 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T06:09:25.792 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:09:25.792 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:09:25.792 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:09:25.792 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:09:25.792 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176784 2024-09-16T06:09:25.793 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176784/$cluster-$name.asok' 2024-09-16T06:09:25.793 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-snaptrim-stats/a '--log-file=td/test-snaptrim-stats/$name.log' '--admin-socket=/tmp/ceph-asok.176784/$cluster-$name.asok' --mon-cluster-log-file=td/test-snaptrim-stats/log --run-dir=td/test-snaptrim-stats '--pid-file=td/test-snaptrim-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-16T06:09:25.995 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T06:09:25.996 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T06:09:25.996 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T06:09:25.997 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T06:09:25.997 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T06:09:25.997 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T06:09:25.997 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T06:09:25.997 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T06:09:25.997 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T06:09:25.997 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:09:25.997 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:09:25.997 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176784 2024-09-16T06:09:25.997 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.176784/ceph-mon.a.asok 2024-09-16T06:09:25.997 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T06:09:25.997 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.176784/ceph-mon.a.asok config get fsid 2024-09-16T06:09:26.077 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T06:09:26.077 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T06:09:26.077 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T06:09:26.077 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T06:09:26.077 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T06:09:26.077 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T06:09:26.077 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T06:09:26.078 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T06:09:26.078 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:09:26.078 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:09:26.078 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176784 2024-09-16T06:09:26.078 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.176784/ceph-mon.a.asok 2024-09-16T06:09:26.078 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T06:09:26.078 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.176784/ceph-mon.a.asok config get mon_host 2024-09-16T06:09:26.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:49: TEST_snaptrim_stats: run_mgr td/test-snaptrim-stats x 2024-09-16T06:09:26.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-snaptrim-stats 2024-09-16T06:09:26.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T06:09:26.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T06:09:26.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T06:09:26.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-snaptrim-stats/x 2024-09-16T06:09:26.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T06:09:27.023 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T06:09:27.023 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:09:27.023 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:09:27.023 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:09:27.023 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:09:27.023 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176784 2024-09-16T06:09:27.024 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176784/$cluster-$name.asok' 2024-09-16T06:09:27.024 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T06:09:27.024 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-snaptrim-stats/x '--log-file=td/test-snaptrim-stats/$name.log' '--admin-socket=/tmp/ceph-asok.176784/$cluster-$name.asok' --run-dir=td/test-snaptrim-stats '--pid-file=td/test-snaptrim-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T06:09:27.051 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: expr 3 - 1 2024-09-16T06:09:27.052 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: seq 0 2 2024-09-16T06:09:27.056 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-16T06:09:27.056 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:52: TEST_snaptrim_stats: run_osd td/test-snaptrim-stats 0 --osd_pool_default_pg_autoscale_mode=off 2024-09-16T06:09:27.056 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-09-16T06:09:27.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T06:09:27.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T06:09:27.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T06:09:27.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/0 2024-09-16T06:09:27.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9b40f4bd-1dcc-4b6e-878d-4e61738f3b4d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-09-16T06:09:27.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:09:27.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:09:27.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:09:27.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/0' 2024-09-16T06:09:27.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/0/journal' 2024-09-16T06:09:27.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T06:09:27.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T06:09:27.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-09-16T06:09:27.058 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T06:09:27.058 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:09:27.058 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:09:27.058 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:09:27.058 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:09:27.058 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176784 2024-09-16T06:09:27.058 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176784/$cluster-$name.asok' 2024-09-16T06:09:27.058 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.176784/$cluster-$name.asok' 2024-09-16T06:09:27.058 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:09:27.058 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T06:09:27.058 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T06:09:27.058 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-09-16T06:09:27.058 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-09-16T06:09:27.059 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:09:27.059 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:09:27.059 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:09:27.059 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:09:27.059 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T06:09:27.059 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-09-16T06:09:27.059 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/0 2024-09-16T06:09:27.059 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T06:09:27.059 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c1b2bccd-c000-4d70-9c66-59ef75d07ae7 2024-09-16T06:09:27.059 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c1b2bccd-c000-4d70-9c66-59ef75d07ae7' 2024-09-16T06:09:27.059 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 c1b2bccd-c000-4d70-9c66-59ef75d07ae7 2024-09-16T06:09:27.060 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T06:09:27.073 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCXy+dmv8pVBBAArrHEuXpUs3LMHxcBUlqWKw== 2024-09-16T06:09:27.073 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCXy+dmv8pVBBAArrHEuXpUs3LMHxcBUlqWKw=="}' 2024-09-16T06:09:27.073 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c1b2bccd-c000-4d70-9c66-59ef75d07ae7 -i td/test-snaptrim-stats/0/new.json 2024-09-16T06:09:27.351 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:09:27.360 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/0/new.json 2024-09-16T06:09:27.361 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=9b40f4bd-1dcc-4b6e-878d-4e61738f3b4d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/0 --osd-journal=td/test-snaptrim-stats/0/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.176784/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQCXy+dmv8pVBBAArrHEuXpUs3LMHxcBUlqWKw== --osd-uuid c1b2bccd-c000-4d70-9c66-59ef75d07ae7 2024-09-16T06:09:27.390 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:09:27.389+0000 7f931331b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:09:27.420 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:09:27.419+0000 7f931331b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:09:27.421 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:09:27.420+0000 7f931331b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:09:27.421 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:09:27.421+0000 7f931331b640 -1 bdev(0x55c339045c00 td/test-snaptrim-stats/0/block) open stat got: (1) Operation not permitted 2024-09-16T06:09:27.421 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:09:27.421+0000 7f931331b640 -1 bluestore(td/test-snaptrim-stats/0) _read_fsid unparsable uuid 2024-09-16T06:09:32.848 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/0/keyring 2024-09-16T06:09:32.848 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T06:09:32.849 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T06:09:32.849 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T06:09:32.849 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T06:09:33.368 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T06:09:33.369 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T06:09:33.369 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=9b40f4bd-1dcc-4b6e-878d-4e61738f3b4d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/0 --osd-journal=td/test-snaptrim-stats/0/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.176784/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-09-16T06:09:33.369 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T06:09:33.369 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T06:09:33.369 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T06:09:33.394 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:09:33.394+0000 7fe0c21cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:09:33.414 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:09:33.413+0000 7fe0c21cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:09:33.415 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:09:33.414+0000 7fe0c21cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:09:33.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T06:09:33.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:09:33.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T06:09:33.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:09:33.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:09:33.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:09:33.795 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:09:33.795 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:09:33.795 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:09:33.795 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:09:34.219 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:09:34.521 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:09:34.521+0000 7fe0c21cc640 -1 Falling back to public interface 2024-09-16T06:09:35.221 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:09:35.221 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:09:35.222 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:09:35.222 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T06:09:35.222 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:09:35.222 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:09:35.636 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:09:36.638 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:09:36.638 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:09:36.638 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:09:36.639 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T06:09:36.639 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:09:36.639 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:09:37.014 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:09:37.366 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:09:37.366+0000 7fe0c21cc640 -1 osd.0 0 log_to_monitors true 2024-09-16T06:09:38.015 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:09:38.015 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:09:38.015 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T06:09:38.016 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T06:09:38.016 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:09:38.016 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:09:38.460 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:09:39.462 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:09:39.462 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:09:39.462 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T06:09:39.463 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T06:09:39.464 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:09:39.464 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:09:39.841 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:09:40.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:09:40.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:09:40.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T06:09:40.843 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T06:09:40.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:09:40.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:09:41.290 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:09:42.292 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:09:42.292 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:09:42.292 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T06:09:42.292 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T06:09:42.293 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:09:42.293 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:09:42.677 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:09:43.678 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:09:43.678 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:09:43.678 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T06:09:43.679 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T06:09:43.679 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:09:43.679 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:09:44.119 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:09:45.120 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:09:45.120 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:09:45.121 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T06:09:45.121 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T06:09:45.121 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:09:45.121 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:09:45.509 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:09:46.510 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:09:46.510 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:09:46.510 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T06:09:46.511 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T06:09:46.511 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:09:46.511 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:09:46.924 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:09:47.926 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:09:47.926 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:09:47.927 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T06:09:47.928 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T06:09:47.928 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:09:47.928 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:09:48.317 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:09:49.318 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:09:49.319 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:09:49.319 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T06:09:49.319 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T06:09:49.319 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:09:49.319 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:09:49.760 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:09:50.762 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:09:50.762 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:09:50.762 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T06:09:50.762 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T06:09:50.763 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:09:50.763 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:09:51.740 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:09:52.739 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:09:52.739 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:09:52.739 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T06:09:52.739 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T06:09:52.740 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:09:52.740 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:09:53.210 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:09:54.211 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:09:54.211 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:09:54.211 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T06:09:54.211 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T06:09:54.212 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:09:54.212 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:09:54.967 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:09:55.969 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:09:55.969 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:09:55.970 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T06:09:55.970 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T06:09:55.970 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:09:55.970 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:09:56.792 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:09:57.793 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:09:57.793 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:09:57.794 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T06:09:57.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T06:09:57.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:09:57.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:09:58.609 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:09:59.611 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:09:59.611 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:09:59.611 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T06:09:59.612 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T06:09:59.612 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:09:59.612 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:10:00.560 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:10:01.561 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:10:01.562 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:10:01.562 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T06:10:01.562 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T06:10:01.562 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:10:01.563 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:10:01.974 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2449012009,v1:127.0.0.1:6803/2449012009] [v2:127.0.0.1:6804/2449012009,v1:127.0.0.1:6805/2449012009] exists,up c1b2bccd-c000-4d70-9c66-59ef75d07ae7 2024-09-16T06:10:01.977 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:10:01.978 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:10:01.978 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:10:01.978 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-16T06:10:01.978 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:52: TEST_snaptrim_stats: run_osd td/test-snaptrim-stats 1 --osd_pool_default_pg_autoscale_mode=off 2024-09-16T06:10:01.978 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-09-16T06:10:01.978 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T06:10:01.978 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T06:10:01.978 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T06:10:01.978 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/1 2024-09-16T06:10:01.978 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9b40f4bd-1dcc-4b6e-878d-4e61738f3b4d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-09-16T06:10:01.978 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:10:01.979 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:10:01.979 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:10:01.979 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/1' 2024-09-16T06:10:01.979 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/1/journal' 2024-09-16T06:10:01.979 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T06:10:01.979 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T06:10:01.979 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-09-16T06:10:01.979 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T06:10:01.979 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:10:01.979 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:10:01.979 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:10:01.979 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:10:01.979 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176784 2024-09-16T06:10:01.980 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176784/$cluster-$name.asok' 2024-09-16T06:10:01.980 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.176784/$cluster-$name.asok' 2024-09-16T06:10:01.980 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:10:01.980 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T06:10:01.980 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T06:10:01.980 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-09-16T06:10:01.980 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-09-16T06:10:01.980 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:10:01.980 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:10:01.980 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:10:01.980 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:10:01.981 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T06:10:01.984 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-09-16T06:10:01.984 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/1 2024-09-16T06:10:01.984 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T06:10:01.984 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d9095609-3ef8-44fc-acd0-1b93573050f9 2024-09-16T06:10:01.984 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d9095609-3ef8-44fc-acd0-1b93573050f9' 2024-09-16T06:10:01.985 INFO:tasks.workunit.client.0.smithi178.stdout:add osd1 d9095609-3ef8-44fc-acd0-1b93573050f9 2024-09-16T06:10:01.985 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T06:10:01.995 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC5y+dmbGZPOxAAte/DItQJQYdTlY+YdTVJFw== 2024-09-16T06:10:01.996 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC5y+dmbGZPOxAAte/DItQJQYdTlY+YdTVJFw=="}' 2024-09-16T06:10:01.996 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d9095609-3ef8-44fc-acd0-1b93573050f9 -i td/test-snaptrim-stats/1/new.json 2024-09-16T06:10:02.478 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:10:02.489 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/1/new.json 2024-09-16T06:10:02.490 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=9b40f4bd-1dcc-4b6e-878d-4e61738f3b4d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/1 --osd-journal=td/test-snaptrim-stats/1/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.176784/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQC5y+dmbGZPOxAAte/DItQJQYdTlY+YdTVJFw== --osd-uuid d9095609-3ef8-44fc-acd0-1b93573050f9 2024-09-16T06:10:02.518 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:10:02.517+0000 7fc7cb811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:10:02.532 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:10:02.531+0000 7fc7cb811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:10:02.532 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:10:02.532+0000 7fc7cb811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:10:02.533 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:10:02.532+0000 7fc7cb811640 -1 bdev(0x5595c87cfc00 td/test-snaptrim-stats/1/block) open stat got: (1) Operation not permitted 2024-09-16T06:10:02.533 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:10:02.532+0000 7fc7cb811640 -1 bluestore(td/test-snaptrim-stats/1) _read_fsid unparsable uuid 2024-09-16T06:10:08.320 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/1/keyring 2024-09-16T06:10:08.320 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T06:10:08.320 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd1 key to auth repository 2024-09-16T06:10:08.321 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T06:10:08.321 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T06:10:08.829 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.1 2024-09-16T06:10:08.830 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T06:10:08.830 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=9b40f4bd-1dcc-4b6e-878d-4e61738f3b4d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/1 --osd-journal=td/test-snaptrim-stats/1/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.176784/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-09-16T06:10:08.830 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T06:10:08.830 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T06:10:08.830 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T06:10:08.855 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:10:08.855+0000 7fdadee14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:10:08.885 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:10:08.884+0000 7fdadee14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:10:08.886 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:10:08.886+0000 7fdadee14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:10:09.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T06:10:09.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:10:09.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T06:10:09.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:10:09.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:10:09.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:10:09.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:10:09.208 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:10:09.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:10:09.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:10:09.613 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:10:10.495 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:10:10.494+0000 7fdadee14640 -1 Falling back to public interface 2024-09-16T06:10:10.614 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:10:10.614 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:10:10.614 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:10:10.615 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T06:10:10.615 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:10:10.615 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:10:10.997 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:10:11.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:10:11.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:10:12.000 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:10:12.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T06:10:12.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:10:12.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:10:12.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:10:13.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:10:13.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:10:13.419 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T06:10:13.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T06:10:13.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:10:13.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:10:13.538 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:10:13.538+0000 7fdadee14640 -1 osd.1 0 log_to_monitors true 2024-09-16T06:10:13.846 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:10:14.847 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:10:14.848 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:10:14.848 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T06:10:14.848 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T06:10:14.848 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:10:14.849 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:10:15.371 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:10:16.373 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:10:16.373 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:10:16.373 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T06:10:16.374 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T06:10:16.374 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:10:16.374 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:10:16.782 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:10:17.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:10:17.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:10:17.784 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T06:10:17.785 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T06:10:17.785 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:10:17.785 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:10:18.165 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:10:19.166 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:10:19.166 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:10:19.167 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T06:10:19.167 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T06:10:19.167 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:10:19.167 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:10:19.670 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:10:20.671 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:10:20.671 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:10:20.672 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T06:10:20.672 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T06:10:20.672 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:10:20.672 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:10:21.109 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:10:22.111 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:10:22.111 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:10:22.111 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T06:10:22.111 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T06:10:22.112 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:10:22.112 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:10:22.493 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:10:23.494 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:10:23.495 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:10:23.495 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T06:10:23.495 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T06:10:23.495 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:10:23.495 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:10:23.932 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:10:24.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:10:24.933 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:10:24.934 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T06:10:24.934 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T06:10:24.934 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:10:24.934 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:10:25.363 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:10:26.364 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:10:26.364 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:10:26.365 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T06:10:26.365 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T06:10:26.365 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:10:26.365 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:10:27.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:10:28.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:10:28.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:10:28.057 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T06:10:28.058 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T06:10:28.058 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:10:28.058 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:10:28.983 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:10:29.984 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:10:30.063 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:10:30.063 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T06:10:30.064 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T06:10:30.064 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:10:30.064 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:10:30.769 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:10:31.770 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:10:31.770 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:10:31.771 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T06:10:31.771 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T06:10:31.771 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:10:31.771 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:10:32.591 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:10:33.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:10:33.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:10:33.593 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T06:10:33.593 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T06:10:33.593 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:10:33.593 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:10:35.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:10:36.045 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:10:36.045 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:10:36.045 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T06:10:36.046 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T06:10:36.046 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:10:36.046 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:10:36.691 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:10:37.692 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:10:37.692 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:10:37.693 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T06:10:37.693 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T06:10:37.693 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:10:37.693 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:10:38.098 INFO:tasks.workunit.client.0.smithi178.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1002975633,v1:127.0.0.1:6811/1002975633] [v2:127.0.0.1:6812/1002975633,v1:127.0.0.1:6813/1002975633] exists,up d9095609-3ef8-44fc-acd0-1b93573050f9 2024-09-16T06:10:38.102 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:10:38.102 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:10:38.102 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:10:38.102 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-16T06:10:38.103 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:52: TEST_snaptrim_stats: run_osd td/test-snaptrim-stats 2 --osd_pool_default_pg_autoscale_mode=off 2024-09-16T06:10:38.103 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-09-16T06:10:38.103 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T06:10:38.103 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T06:10:38.103 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T06:10:38.103 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/2 2024-09-16T06:10:38.103 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9b40f4bd-1dcc-4b6e-878d-4e61738f3b4d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-09-16T06:10:38.103 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:10:38.103 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:10:38.103 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:10:38.103 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/2' 2024-09-16T06:10:38.103 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/2/journal' 2024-09-16T06:10:38.104 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T06:10:38.104 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T06:10:38.104 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-09-16T06:10:38.104 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T06:10:38.104 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:10:38.105 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:10:38.105 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:10:38.105 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:10:38.105 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176784 2024-09-16T06:10:38.105 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176784/$cluster-$name.asok' 2024-09-16T06:10:38.105 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.176784/$cluster-$name.asok' 2024-09-16T06:10:38.105 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:10:38.105 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T06:10:38.105 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T06:10:38.105 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-09-16T06:10:38.105 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-09-16T06:10:38.105 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:10:38.106 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:10:38.106 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:10:38.106 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:10:38.106 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T06:10:38.106 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-09-16T06:10:38.106 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/2 2024-09-16T06:10:38.106 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T06:10:38.106 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c25d2e94-142c-491e-a6b0-1fd4687ac29c 2024-09-16T06:10:38.106 INFO:tasks.workunit.client.0.smithi178.stdout:add osd2 c25d2e94-142c-491e-a6b0-1fd4687ac29c 2024-09-16T06:10:38.107 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 c25d2e94-142c-491e-a6b0-1fd4687ac29c' 2024-09-16T06:10:38.107 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T06:10:38.119 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDey+dmNf4QBxAAKFezF2QHF9R5Cck+h9fCMA== 2024-09-16T06:10:38.119 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDey+dmNf4QBxAAKFezF2QHF9R5Cck+h9fCMA=="}' 2024-09-16T06:10:38.119 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c25d2e94-142c-491e-a6b0-1fd4687ac29c -i td/test-snaptrim-stats/2/new.json 2024-09-16T06:10:38.674 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:10:38.686 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/2/new.json 2024-09-16T06:10:38.686 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=9b40f4bd-1dcc-4b6e-878d-4e61738f3b4d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/2 --osd-journal=td/test-snaptrim-stats/2/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.176784/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQDey+dmNf4QBxAAKFezF2QHF9R5Cck+h9fCMA== --osd-uuid c25d2e94-142c-491e-a6b0-1fd4687ac29c 2024-09-16T06:10:38.715 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:10:38.714+0000 7fb104411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:10:38.732 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:10:38.732+0000 7fb104411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:10:38.733 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:10:38.733+0000 7fb104411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:10:38.734 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:10:38.733+0000 7fb104411640 -1 bdev(0x55c3369f1c00 td/test-snaptrim-stats/2/block) open stat got: (1) Operation not permitted 2024-09-16T06:10:38.734 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:10:38.733+0000 7fb104411640 -1 bluestore(td/test-snaptrim-stats/2) _read_fsid unparsable uuid 2024-09-16T06:10:44.906 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/2/keyring 2024-09-16T06:10:44.906 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T06:10:44.907 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T06:10:44.907 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd2 key to auth repository 2024-09-16T06:10:44.908 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T06:10:45.419 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.2 2024-09-16T06:10:45.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T06:10:45.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=9b40f4bd-1dcc-4b6e-878d-4e61738f3b4d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/2 --osd-journal=td/test-snaptrim-stats/2/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.176784/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-09-16T06:10:45.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T06:10:45.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T06:10:45.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T06:10:45.446 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:10:45.445+0000 7ff4f83db640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:10:45.447 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:10:45.447+0000 7ff4f83db640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:10:45.449 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:10:45.448+0000 7ff4f83db640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:10:45.829 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T06:10:45.829 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:10:45.829 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T06:10:45.829 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:10:45.829 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:10:45.830 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:10:45.830 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:10:45.830 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:10:45.830 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:10:45.830 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:10:46.197 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:10:47.198 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:10:47.198 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:10:47.199 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:10:47.199 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T06:10:47.199 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:10:47.199 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:10:47.287 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:10:47.287+0000 7ff4f83db640 -1 Falling back to public interface 2024-09-16T06:10:47.623 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:10:48.625 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:10:48.625 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:10:48.625 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:10:48.626 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T06:10:48.626 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:10:48.626 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:10:49.004 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:10:50.005 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:10:50.005 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:10:50.005 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T06:10:50.006 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T06:10:50.006 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:10:50.006 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:10:50.164 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:10:50.163+0000 7ff4f83db640 -1 osd.2 0 log_to_monitors true 2024-09-16T06:10:50.386 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:10:51.387 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:10:51.387 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:10:51.388 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T06:10:51.388 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T06:10:51.389 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:10:51.389 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:10:51.876 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:10:52.877 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:10:52.877 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:10:52.878 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T06:10:52.878 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T06:10:52.878 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:10:52.878 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:10:53.438 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:10:54.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:10:54.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:10:54.440 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T06:10:54.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T06:10:54.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:10:54.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:10:54.820 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:10:55.821 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:10:55.822 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:10:55.822 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T06:10:55.822 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T06:10:55.822 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:10:55.823 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:10:56.222 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:10:57.224 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:10:57.224 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:10:57.224 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T06:10:57.225 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T06:10:57.225 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:10:57.225 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:10:57.713 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:10:58.715 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:10:58.715 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:10:58.715 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T06:10:58.716 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T06:10:58.716 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:10:58.716 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:10:59.239 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:11:00.240 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:11:00.241 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:11:00.241 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T06:11:00.241 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T06:11:00.242 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:11:00.242 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:11:00.660 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:11:01.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:11:01.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:11:01.662 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T06:11:01.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T06:11:01.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:11:01.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:11:02.126 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:11:03.127 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:11:03.127 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:11:03.127 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T06:11:03.128 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T06:11:03.128 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:11:03.128 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:11:03.688 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:11:04.689 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:11:04.689 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:11:04.690 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T06:11:04.690 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T06:11:04.690 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:11:04.690 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:11:05.642 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:11:06.643 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:11:06.643 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:11:06.644 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T06:11:06.644 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T06:11:06.644 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:11:06.644 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:11:07.557 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:11:08.558 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:11:08.558 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:11:08.559 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T06:11:08.559 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T06:11:08.559 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:11:08.559 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:11:09.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:11:10.308 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:11:10.309 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:11:10.309 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T06:11:10.309 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T06:11:10.310 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:11:10.310 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:11:11.494 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:11:12.495 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:11:12.495 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:11:12.495 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T06:11:12.495 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T06:11:12.496 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:11:12.496 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:11:13.267 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:11:14.268 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:11:14.268 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:11:14.268 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T06:11:14.268 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T06:11:14.269 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:11:14.269 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:11:14.714 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:11:15.715 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:11:15.716 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:11:15.716 INFO:tasks.workunit.client.0.smithi178.stdout:19 2024-09-16T06:11:15.716 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-16T06:11:15.716 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:11:15.717 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:11:16.086 INFO:tasks.workunit.client.0.smithi178.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3357307998,v1:127.0.0.1:6819/3357307998] [v2:127.0.0.1:6820/3357307998,v1:127.0.0.1:6821/3357307998] exists,up c25d2e94-142c-491e-a6b0-1fd4687ac29c 2024-09-16T06:11:16.086 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:11:16.087 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:11:16.087 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:11:16.087 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:56: TEST_snaptrim_stats: ceph osd set noscrub 2024-09-16T06:11:16.688 INFO:tasks.workunit.client.0.smithi178.stderr:noscrub is set 2024-09-16T06:11:16.699 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:57: TEST_snaptrim_stats: ceph osd set nodeep-scrub 2024-09-16T06:11:17.434 INFO:tasks.workunit.client.0.smithi178.stderr:nodeep-scrub is set 2024-09-16T06:11:17.446 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:60: TEST_snaptrim_stats: create_pool test 8 8 2024-09-16T06:11:17.446 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 8 8 2024-09-16T06:11:17.980 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'test' already exists 2024-09-16T06:11:17.992 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T06:11:18.994 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:61: TEST_snaptrim_stats: wait_for_clean 2024-09-16T06:11:18.994 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T06:11:18.994 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T06:11:18.994 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T06:11:18.994 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T06:11:18.996 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T06:11:18.996 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T06:11:18.996 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T06:11:18.996 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T06:11:18.996 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T06:11:19.090 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T06:11:19.090 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T06:11:19.090 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T06:11:19.090 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T06:11:19.090 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-16T06:11:19.091 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-16T06:11:19.634 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-16T06:11:19.634 INFO:tasks.workunit.client.0.smithi178.stderr:1 2024-09-16T06:11:19.634 INFO:tasks.workunit.client.0.smithi178.stderr:2' 2024-09-16T06:11:19.634 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-16T06:11:19.634 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T06:11:19.634 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T06:11:19.776 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836504 2024-09-16T06:11:19.776 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836504 2024-09-16T06:11:19.776 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504' 2024-09-16T06:11:19.776 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T06:11:19.776 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T06:11:19.895 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-09-16T06:11:19.895 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-09-16T06:11:19.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-42949672976' 2024-09-16T06:11:19.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T06:11:19.896 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T06:11:20.012 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509449 2024-09-16T06:11:20.012 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509449 2024-09-16T06:11:20.012 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-42949672976 2-64424509449' 2024-09-16T06:11:20.012 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T06:11:20.013 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836504 2024-09-16T06:11:20.013 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T06:11:20.013 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-16T06:11:20.014 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836504 2024-09-16T06:11:20.014 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T06:11:20.015 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836504 2024-09-16T06:11:20.015 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.0 seq 21474836504 2024-09-16T06:11:20.016 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-09-16T06:11:20.016 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T06:11:20.380 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836504 2024-09-16T06:11:20.380 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-16T06:11:21.381 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T06:11:21.381 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T06:11:21.757 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836504 2024-09-16T06:11:21.757 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T06:11:21.758 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-09-16T06:11:21.758 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T06:11:21.758 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-09-16T06:11:21.759 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-09-16T06:11:21.759 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T06:11:21.760 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-09-16T06:11:21.760 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.1 seq 42949672976 2024-09-16T06:11:21.761 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-09-16T06:11:21.761 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T06:11:22.129 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-09-16T06:11:22.129 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T06:11:22.130 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509449 2024-09-16T06:11:22.130 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T06:11:22.130 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-09-16T06:11:22.131 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509449 2024-09-16T06:11:22.131 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T06:11:22.132 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509449 2024-09-16T06:11:22.132 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.2 seq 64424509449 2024-09-16T06:11:22.133 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-09-16T06:11:22.133 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T06:11:22.502 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509450 -lt 64424509449 2024-09-16T06:11:22.503 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T06:11:22.503 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T06:11:22.503 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T06:11:23.047 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-09-16T06:11:23.047 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T06:11:23.047 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T06:11:23.047 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T06:11:23.047 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T06:11:23.048 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T06:11:23.048 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T06:11:23.048 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T06:11:23.412 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-16T06:11:23.413 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T06:11:23.413 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T06:11:23.413 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T06:11:23.907 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-09-16T06:11:23.907 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T06:11:23.907 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T06:11:23.908 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:62: TEST_snaptrim_stats: ceph osd dump 2024-09-16T06:11:23.908 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:62: TEST_snaptrim_stats: grep '^pool.*['\'']test['\'']' 2024-09-16T06:11:23.908 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:62: TEST_snaptrim_stats: awk '{ print $2 }' 2024-09-16T06:11:24.314 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:62: TEST_snaptrim_stats: poolid=1 2024-09-16T06:11:24.314 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:65: TEST_snaptrim_stats: TESTDATA=testdata.1 2024-09-16T06:11:24.314 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:66: TEST_snaptrim_stats: dd if=/dev/urandom of=testdata.1 bs=4096 count=1 2024-09-16T06:11:24.314 INFO:tasks.workunit.client.0.smithi178.stderr:1+0 records in 2024-09-16T06:11:24.315 INFO:tasks.workunit.client.0.smithi178.stderr:1+0 records out 2024-09-16T06:11:24.315 INFO:tasks.workunit.client.0.smithi178.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 5.3251e-05 s, 76.9 MB/s 2024-09-16T06:11:24.315 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: seq 1 10 2024-09-16T06:11:24.316 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T06:11:24.316 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj1 testdata.1 2024-09-16T06:11:24.490 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T06:11:24.490 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj2 testdata.1 2024-09-16T06:11:24.692 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T06:11:24.692 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj3 testdata.1 2024-09-16T06:11:24.826 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T06:11:24.826 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj4 testdata.1 2024-09-16T06:11:24.968 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T06:11:24.968 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj5 testdata.1 2024-09-16T06:11:25.110 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T06:11:25.110 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj6 testdata.1 2024-09-16T06:11:25.245 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T06:11:25.245 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj7 testdata.1 2024-09-16T06:11:25.380 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T06:11:25.380 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj8 testdata.1 2024-09-16T06:11:25.513 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T06:11:25.513 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj9 testdata.1 2024-09-16T06:11:25.647 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T06:11:25.647 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj10 testdata.1 2024-09-16T06:11:25.773 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:71: TEST_snaptrim_stats: rm -f testdata.1 2024-09-16T06:11:25.773 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:74: TEST_snaptrim_stats: SNAP=1 2024-09-16T06:11:25.774 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:75: TEST_snaptrim_stats: rados -p test mksnap snap1 2024-09-16T06:11:25.923 INFO:tasks.workunit.client.0.smithi178.stdout:created pool test snap snap1 2024-09-16T06:11:25.925 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:76: TEST_snaptrim_stats: TESTDATA=testdata.2 2024-09-16T06:11:25.925 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:77: TEST_snaptrim_stats: dd if=/dev/urandom of=testdata.2 bs=4096 count=1 2024-09-16T06:11:25.926 INFO:tasks.workunit.client.0.smithi178.stderr:1+0 records in 2024-09-16T06:11:25.926 INFO:tasks.workunit.client.0.smithi178.stderr:1+0 records out 2024-09-16T06:11:25.926 INFO:tasks.workunit.client.0.smithi178.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 4.7928e-05 s, 85.5 MB/s 2024-09-16T06:11:25.926 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: seq 1 10 2024-09-16T06:11:25.927 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T06:11:25.927 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj1 testdata.2 2024-09-16T06:11:26.274 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T06:11:26.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj2 testdata.2 2024-09-16T06:11:26.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T06:11:26.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj3 testdata.2 2024-09-16T06:11:26.552 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T06:11:26.552 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj4 testdata.2 2024-09-16T06:11:26.694 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T06:11:26.695 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj5 testdata.2 2024-09-16T06:11:26.820 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T06:11:26.820 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj6 testdata.2 2024-09-16T06:11:26.971 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T06:11:26.972 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj7 testdata.2 2024-09-16T06:11:27.130 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T06:11:27.130 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj8 testdata.2 2024-09-16T06:11:27.315 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T06:11:27.315 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj9 testdata.2 2024-09-16T06:11:27.457 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T06:11:27.457 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj10 testdata.2 2024-09-16T06:11:27.583 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:82: TEST_snaptrim_stats: rm -f testdata.2 2024-09-16T06:11:27.584 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:85: TEST_snaptrim_stats: rados -p test rmsnap snap1 2024-09-16T06:11:27.733 INFO:tasks.workunit.client.0.smithi178.stdout:removed pool test snap snap1 2024-09-16T06:11:27.735 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:88: TEST_snaptrim_stats: wait_for_clean 2024-09-16T06:11:27.735 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T06:11:27.735 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T06:11:27.735 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T06:11:27.736 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T06:11:27.736 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T06:11:27.737 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T06:11:27.737 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T06:11:27.737 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T06:11:27.737 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T06:11:27.822 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T06:11:27.822 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T06:11:27.822 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T06:11:27.822 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T06:11:27.822 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-16T06:11:27.822 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-16T06:11:28.202 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-16T06:11:28.202 INFO:tasks.workunit.client.0.smithi178.stderr:1 2024-09-16T06:11:28.202 INFO:tasks.workunit.client.0.smithi178.stderr:2' 2024-09-16T06:11:28.202 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-16T06:11:28.202 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T06:11:28.203 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T06:11:28.312 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836508 2024-09-16T06:11:28.312 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836508 2024-09-16T06:11:28.312 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508' 2024-09-16T06:11:28.313 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T06:11:28.313 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T06:11:28.422 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672980 2024-09-16T06:11:28.423 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672980 2024-09-16T06:11:28.423 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672980' 2024-09-16T06:11:28.423 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T06:11:28.423 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T06:11:28.697 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-09-16T06:11:28.697 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-09-16T06:11:28.697 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672980 2-64424509453' 2024-09-16T06:11:28.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T06:11:28.698 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836508 2024-09-16T06:11:28.698 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T06:11:28.699 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-16T06:11:28.699 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836508 2024-09-16T06:11:28.699 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T06:11:28.700 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836508 2024-09-16T06:11:28.701 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.0 seq 21474836508 2024-09-16T06:11:28.701 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-09-16T06:11:28.701 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T06:11:29.111 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836508 2024-09-16T06:11:29.112 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T06:11:29.112 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672980 2024-09-16T06:11:29.112 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T06:11:29.113 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-09-16T06:11:29.113 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672980 2024-09-16T06:11:29.113 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T06:11:29.114 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672980 2024-09-16T06:11:29.114 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-09-16T06:11:29.115 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.1 seq 42949672980 2024-09-16T06:11:29.115 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T06:11:29.671 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672980 -lt 42949672980 2024-09-16T06:11:29.671 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T06:11:29.672 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-09-16T06:11:29.672 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T06:11:29.673 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-09-16T06:11:29.673 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-09-16T06:11:29.673 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T06:11:29.674 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-09-16T06:11:29.674 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-09-16T06:11:29.675 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.2 seq 64424509453 2024-09-16T06:11:29.675 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T06:11:30.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509453 2024-09-16T06:11:30.044 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T06:11:30.044 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T06:11:30.044 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T06:11:30.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-09-16T06:11:30.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T06:11:30.544 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T06:11:30.544 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T06:11:30.544 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T06:11:30.544 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T06:11:30.544 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T06:11:30.544 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T06:11:30.960 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-16T06:11:30.961 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T06:11:30.964 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T06:11:30.964 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T06:11:31.458 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-09-16T06:11:31.458 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T06:11:31.459 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T06:11:31.459 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:89: TEST_snaptrim_stats: sleep 10 2024-09-16T06:11:41.460 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:90: TEST_snaptrim_stats: local objects_trimmed=0 2024-09-16T06:11:41.460 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:91: TEST_snaptrim_stats: local snaptrim_duration_total=0.0 2024-09-16T06:11:41.460 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: expr 8 - 1 2024-09-16T06:11:41.461 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: seq 0 7 2024-09-16T06:11:41.462 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T06:11:41.463 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.0 2024-09-16T06:11:41.464 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.0 query 2024-09-16T06:11:41.464 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-09-16T06:11:41.591 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 0 + 2 2024-09-16T06:11:41.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=2 2024-09-16T06:11:41.593 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-09-16T06:11:41.593 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.0 query 2024-09-16T06:11:41.593 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-09-16T06:11:41.709 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 0.0 + 0.81454520100000005 2024-09-16T06:11:41.709 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=.81454520100000005 2024-09-16T06:11:41.710 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T06:11:41.710 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.1 2024-09-16T06:11:41.710 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.1 query 2024-09-16T06:11:41.710 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-09-16T06:11:41.827 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 2 + 2 2024-09-16T06:11:41.828 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=4 2024-09-16T06:11:41.828 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-09-16T06:11:41.829 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.1 query 2024-09-16T06:11:41.829 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-09-16T06:11:41.944 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo .81454520100000005 + 0.52194195499999996 2024-09-16T06:11:41.945 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=1.33648715600000001 2024-09-16T06:11:41.945 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T06:11:41.945 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.2 2024-09-16T06:11:41.945 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.2 query 2024-09-16T06:11:41.945 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-09-16T06:11:42.061 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 4 + 0 2024-09-16T06:11:42.061 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=4 2024-09-16T06:11:42.062 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-09-16T06:11:42.063 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.2 query 2024-09-16T06:11:42.063 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-09-16T06:11:42.179 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 1.33648715600000001 + 0.00035284699999999999 2024-09-16T06:11:42.180 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=1.33684000300000000999 2024-09-16T06:11:42.180 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T06:11:42.180 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.3 2024-09-16T06:11:42.181 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.3 query 2024-09-16T06:11:42.181 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-09-16T06:11:42.296 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 4 + 2 2024-09-16T06:11:42.297 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=6 2024-09-16T06:11:42.298 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-09-16T06:11:42.298 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.3 query 2024-09-16T06:11:42.298 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-09-16T06:11:42.415 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 1.33684000300000000999 + 0.46147531800000002 2024-09-16T06:11:42.415 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=1.79831532100000002999 2024-09-16T06:11:42.415 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T06:11:42.415 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.4 2024-09-16T06:11:42.416 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.4 query 2024-09-16T06:11:42.416 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-09-16T06:11:42.530 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 6 + 1 2024-09-16T06:11:42.531 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=7 2024-09-16T06:11:42.532 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-09-16T06:11:42.532 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.4 query 2024-09-16T06:11:42.532 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-09-16T06:11:42.647 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 1.79831532100000002999 + 0.26099076900000001 2024-09-16T06:11:42.648 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=2.05930609000000003999 2024-09-16T06:11:42.648 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T06:11:42.648 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.5 2024-09-16T06:11:42.648 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.5 query 2024-09-16T06:11:42.649 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-09-16T06:11:42.764 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 7 + 1 2024-09-16T06:11:42.765 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=8 2024-09-16T06:11:42.766 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-09-16T06:11:42.767 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.5 query 2024-09-16T06:11:42.767 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-09-16T06:11:42.882 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 2.05930609000000003999 + 0.52171467000000005 2024-09-16T06:11:42.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=2.58102076000000008999 2024-09-16T06:11:42.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T06:11:42.883 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.6 2024-09-16T06:11:42.883 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.6 query 2024-09-16T06:11:42.883 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-09-16T06:11:42.999 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 8 + 0 2024-09-16T06:11:43.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=8 2024-09-16T06:11:43.001 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-09-16T06:11:43.002 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.6 query 2024-09-16T06:11:43.002 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-09-16T06:11:43.147 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 2.58102076000000008999 + 0.58720929099999997 2024-09-16T06:11:43.148 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=3.16823005100000005999 2024-09-16T06:11:43.148 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T06:11:43.148 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.7 2024-09-16T06:11:43.148 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.7 query 2024-09-16T06:11:43.149 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-09-16T06:11:43.265 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 8 + 2 2024-09-16T06:11:43.266 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=10 2024-09-16T06:11:43.267 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-09-16T06:11:43.267 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.7 query 2024-09-16T06:11:43.267 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-09-16T06:11:43.382 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 3.16823005100000005999 + 1.0485740080000001 2024-09-16T06:11:43.383 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=4.21680405900000015999 2024-09-16T06:11:43.383 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:100: TEST_snaptrim_stats: test 10 -eq 10 2024-09-16T06:11:43.383 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:101: TEST_snaptrim_stats: echo '4.21680405900000015999 > 0.0' 2024-09-16T06:11:43.383 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:101: TEST_snaptrim_stats: bc 2024-09-16T06:11:43.383 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:11:43.384 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:103: TEST_snaptrim_stats: teardown td/test-snaptrim-stats 2024-09-16T06:11:43.385 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-09-16T06:11:43.385 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T06:11:43.385 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-09-16T06:11:43.385 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:11:43.385 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:11:43.385 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:11:43.385 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:11:43.385 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:11:43.491 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:11:43.492 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:11:43.492 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:11:43.493 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:11:43.494 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:11:43.494 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:11:43.494 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:11:43.494 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:11:43.495 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:11:43.495 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:11:43.495 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:11:43.496 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:11:43.497 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T06:11:43.497 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-09-16T06:11:43.527 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:11:43.528 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:11:43.528 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176784 2024-09-16T06:11:43.528 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.176784 2024-09-16T06:11:43.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:11:43.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:11:43.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:34: run: teardown td/test-snaptrim-stats 2024-09-16T06:11:43.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-09-16T06:11:43.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T06:11:43.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-09-16T06:11:43.530 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:11:43.530 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:11:43.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:11:43.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:11:43.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:11:43.531 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:11:43.531 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:11:43.531 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:11:43.532 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:11:43.533 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:11:43.533 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:11:43.533 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:11:43.534 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:11:43.534 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:11:43.534 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:11:43.534 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:11:43.535 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:11:43.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T06:11:43.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-09-16T06:11:43.537 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:11:43.537 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:11:43.537 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176784 2024-09-16T06:11:43.537 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.176784 2024-09-16T06:11:43.540 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:11:43.540 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:11:43.540 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:31: run: for func in $funcs 2024-09-16T06:11:43.540 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:32: run: setup td/test-snaptrim-stats 2024-09-16T06:11:43.540 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-snaptrim-stats 2024-09-16T06:11:43.540 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-snaptrim-stats 2024-09-16T06:11:43.540 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-09-16T06:11:43.540 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T06:11:43.540 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-09-16T06:11:43.540 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:11:43.540 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:11:43.540 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:11:43.541 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:11:43.541 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:11:43.541 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:11:43.541 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:11:43.541 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:11:43.542 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:11:43.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:11:43.543 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:11:43.543 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:11:43.544 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:11:43.544 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:11:43.544 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:11:43.544 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:11:43.544 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:11:43.545 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T06:11:43.545 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-09-16T06:11:43.547 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:11:43.547 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:11:43.547 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176784 2024-09-16T06:11:43.547 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.176784 2024-09-16T06:11:43.547 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:11:43.548 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:11:43.548 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-snaptrim-stats 2024-09-16T06:11:43.549 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T06:11:43.549 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:11:43.549 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176784 2024-09-16T06:11:43.549 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.176784 2024-09-16T06:11:43.552 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T06:11:43.552 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T06:11:43.552 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T06:11:43.552 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-snaptrim-stats 1' TERM HUP INT 2024-09-16T06:11:43.552 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:33: run: TEST_snaptrim_stats_multiple_snaps td/test-snaptrim-stats 2024-09-16T06:11:43.552 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:107: TEST_snaptrim_stats_multiple_snaps: local dir=td/test-snaptrim-stats 2024-09-16T06:11:43.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:108: TEST_snaptrim_stats_multiple_snaps: local poolname=test 2024-09-16T06:11:43.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:109: TEST_snaptrim_stats_multiple_snaps: local OSDS=3 2024-09-16T06:11:43.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:110: TEST_snaptrim_stats_multiple_snaps: local PGNUM=8 2024-09-16T06:11:43.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:111: TEST_snaptrim_stats_multiple_snaps: local PGPNUM=8 2024-09-16T06:11:43.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:112: TEST_snaptrim_stats_multiple_snaps: local objects=10 2024-09-16T06:11:43.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:113: TEST_snaptrim_stats_multiple_snaps: local WAIT_FOR_UPDATE=10 2024-09-16T06:11:43.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:115: TEST_snaptrim_stats_multiple_snaps: setup td/test-snaptrim-stats 2024-09-16T06:11:43.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-snaptrim-stats 2024-09-16T06:11:43.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-snaptrim-stats 2024-09-16T06:11:43.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-09-16T06:11:43.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T06:11:43.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-09-16T06:11:43.553 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:11:43.553 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:11:43.554 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:11:43.554 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:11:43.554 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:11:43.554 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:11:43.554 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:11:43.554 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:11:43.555 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:11:43.555 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:11:43.555 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:11:43.555 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:11:43.556 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:11:43.556 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:11:43.556 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:11:43.556 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:11:43.557 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:11:43.558 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T06:11:43.558 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-09-16T06:11:43.559 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:11:43.559 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:11:43.559 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176784 2024-09-16T06:11:43.559 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.176784 2024-09-16T06:11:43.560 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:11:43.560 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:11:43.560 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-snaptrim-stats 2024-09-16T06:11:43.561 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T06:11:43.561 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:11:43.561 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176784 2024-09-16T06:11:43.561 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.176784 2024-09-16T06:11:43.563 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T06:11:43.563 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T06:11:43.563 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T06:11:43.564 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-snaptrim-stats 1' TERM HUP INT 2024-09-16T06:11:43.564 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:116: TEST_snaptrim_stats_multiple_snaps: run_mon td/test-snaptrim-stats a --osd_pool_default_size=3 2024-09-16T06:11:43.564 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-snaptrim-stats 2024-09-16T06:11:43.564 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T06:11:43.564 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T06:11:43.564 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T06:11:43.564 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-snaptrim-stats/a 2024-09-16T06:11:43.564 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-snaptrim-stats/a --run-dir=td/test-snaptrim-stats --osd_pool_default_size=3 2024-09-16T06:11:43.946 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T06:11:43.947 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:11:43.947 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:11:43.947 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:11:43.947 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:11:43.947 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176784 2024-09-16T06:11:43.947 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176784/$cluster-$name.asok' 2024-09-16T06:11:43.947 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-snaptrim-stats/a '--log-file=td/test-snaptrim-stats/$name.log' '--admin-socket=/tmp/ceph-asok.176784/$cluster-$name.asok' --mon-cluster-log-file=td/test-snaptrim-stats/log --run-dir=td/test-snaptrim-stats '--pid-file=td/test-snaptrim-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-16T06:11:44.130 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T06:11:44.132 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T06:11:44.132 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T06:11:44.132 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T06:11:44.132 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T06:11:44.132 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T06:11:44.132 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T06:11:44.132 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T06:11:44.132 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T06:11:44.133 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:11:44.133 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:11:44.133 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176784 2024-09-16T06:11:44.133 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.176784/ceph-mon.a.asok 2024-09-16T06:11:44.133 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T06:11:44.133 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.176784/ceph-mon.a.asok config get fsid 2024-09-16T06:11:44.213 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T06:11:44.213 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T06:11:44.213 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T06:11:44.213 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T06:11:44.213 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T06:11:44.213 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T06:11:44.213 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T06:11:44.213 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T06:11:44.213 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:11:44.213 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:11:44.214 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176784 2024-09-16T06:11:44.214 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.176784/ceph-mon.a.asok 2024-09-16T06:11:44.214 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T06:11:44.214 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.176784/ceph-mon.a.asok config get mon_host 2024-09-16T06:11:44.291 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:117: TEST_snaptrim_stats_multiple_snaps: run_mgr td/test-snaptrim-stats x 2024-09-16T06:11:44.291 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-snaptrim-stats 2024-09-16T06:11:44.291 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T06:11:44.291 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T06:11:44.292 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T06:11:44.292 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-snaptrim-stats/x 2024-09-16T06:11:44.292 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T06:11:45.135 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T06:11:45.135 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:11:45.135 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:11:45.135 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:11:45.135 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:11:45.135 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176784 2024-09-16T06:11:45.135 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176784/$cluster-$name.asok' 2024-09-16T06:11:45.136 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T06:11:45.136 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-snaptrim-stats/x '--log-file=td/test-snaptrim-stats/$name.log' '--admin-socket=/tmp/ceph-asok.176784/$cluster-$name.asok' --run-dir=td/test-snaptrim-stats '--pid-file=td/test-snaptrim-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T06:11:45.162 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: expr 3 - 1 2024-09-16T06:11:45.163 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: seq 0 2 2024-09-16T06:11:45.166 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-16T06:11:45.167 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:120: TEST_snaptrim_stats_multiple_snaps: run_osd td/test-snaptrim-stats 0 --osd_pool_default_pg_autoscale_mode=off 2024-09-16T06:11:45.167 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-09-16T06:11:45.167 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T06:11:45.167 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T06:11:45.167 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T06:11:45.167 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/0 2024-09-16T06:11:45.167 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9b40f4bd-1dcc-4b6e-878d-4e61738f3b4d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-09-16T06:11:45.167 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:11:45.167 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:11:45.167 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:11:45.167 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/0' 2024-09-16T06:11:45.167 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/0/journal' 2024-09-16T06:11:45.168 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T06:11:45.168 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T06:11:45.168 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-09-16T06:11:45.168 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T06:11:45.168 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:11:45.168 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:11:45.168 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:11:45.168 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:11:45.168 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176784 2024-09-16T06:11:45.168 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176784/$cluster-$name.asok' 2024-09-16T06:11:45.168 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.176784/$cluster-$name.asok' 2024-09-16T06:11:45.168 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:11:45.168 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T06:11:45.168 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T06:11:45.169 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-09-16T06:11:45.169 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-09-16T06:11:45.169 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:11:45.169 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:11:45.169 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:11:45.169 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:11:45.169 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T06:11:45.169 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-09-16T06:11:45.169 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/0 2024-09-16T06:11:45.169 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T06:11:45.170 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=67307176-bf4f-4fd6-bac2-9f1077a7409f 2024-09-16T06:11:45.170 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 67307176-bf4f-4fd6-bac2-9f1077a7409f 2024-09-16T06:11:45.170 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 67307176-bf4f-4fd6-bac2-9f1077a7409f' 2024-09-16T06:11:45.170 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T06:11:45.185 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAhzOdmReH9ChAA+5t6RK00Zo3vZyK0yl7ofg== 2024-09-16T06:11:45.185 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAhzOdmReH9ChAA+5t6RK00Zo3vZyK0yl7ofg=="}' 2024-09-16T06:11:45.185 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 67307176-bf4f-4fd6-bac2-9f1077a7409f -i td/test-snaptrim-stats/0/new.json 2024-09-16T06:11:45.447 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:11:45.456 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/0/new.json 2024-09-16T06:11:45.457 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=9b40f4bd-1dcc-4b6e-878d-4e61738f3b4d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/0 --osd-journal=td/test-snaptrim-stats/0/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.176784/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQAhzOdmReH9ChAA+5t6RK00Zo3vZyK0yl7ofg== --osd-uuid 67307176-bf4f-4fd6-bac2-9f1077a7409f 2024-09-16T06:11:45.486 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:11:45.485+0000 7f827981e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:11:45.515 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:11:45.515+0000 7f827981e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:11:45.516 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:11:45.516+0000 7f827981e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:11:45.517 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:11:45.516+0000 7f827981e640 -1 bdev(0x5652ed55bc00 td/test-snaptrim-stats/0/block) open stat got: (1) Operation not permitted 2024-09-16T06:11:45.517 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:11:45.516+0000 7f827981e640 -1 bluestore(td/test-snaptrim-stats/0) _read_fsid unparsable uuid 2024-09-16T06:11:51.105 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/0/keyring 2024-09-16T06:11:51.105 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T06:11:51.106 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T06:11:51.106 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T06:11:51.106 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T06:11:51.618 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T06:11:51.618 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T06:11:51.619 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=9b40f4bd-1dcc-4b6e-878d-4e61738f3b4d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/0 --osd-journal=td/test-snaptrim-stats/0/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.176784/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-09-16T06:11:51.619 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T06:11:51.619 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T06:11:51.619 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T06:11:51.644 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:11:51.644+0000 7f5705a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:11:51.677 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:11:51.676+0000 7f5705a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:11:51.678 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:11:51.677+0000 7f5705a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:11:51.987 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T06:11:51.987 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:11:51.987 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T06:11:51.987 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:11:51.988 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:11:51.988 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:11:51.988 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:11:51.988 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:11:51.988 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:11:51.989 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:11:52.390 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:11:52.770 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:11:52.769+0000 7f5705a0f640 -1 Falling back to public interface 2024-09-16T06:11:53.391 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:11:53.392 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:11:53.392 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:11:53.392 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T06:11:53.393 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:11:53.393 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:11:53.808 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:11:54.809 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:11:54.810 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:11:54.810 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:11:54.810 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T06:11:54.811 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:11:54.811 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:11:55.189 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:11:55.597 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:11:55.596+0000 7f5705a0f640 -1 osd.0 0 log_to_monitors true 2024-09-16T06:11:56.191 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:11:56.191 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:11:56.191 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T06:11:56.192 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T06:11:56.192 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:11:56.192 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:11:56.662 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:11:57.663 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:11:57.663 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:11:57.664 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T06:11:57.664 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T06:11:57.664 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:11:57.664 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:11:58.087 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:11:59.088 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:11:59.089 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:11:59.089 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T06:11:59.089 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T06:11:59.089 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:11:59.090 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:11:59.503 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:12:00.504 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:12:00.505 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:12:00.505 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T06:12:00.506 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T06:12:00.506 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:12:00.506 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:12:00.904 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:12:01.905 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:12:01.905 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:12:01.906 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T06:12:01.906 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T06:12:01.906 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:12:01.906 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:12:02.330 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:12:03.331 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:12:03.332 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T06:12:03.332 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:12:03.333 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T06:12:03.333 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:12:03.333 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:12:03.735 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:12:04.737 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:12:04.737 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:12:04.737 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T06:12:04.738 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T06:12:04.738 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:12:04.738 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:12:05.137 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:12:06.138 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:12:06.138 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:12:06.138 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T06:12:06.139 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T06:12:06.139 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:12:06.139 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:12:06.519 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:12:07.521 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:12:07.521 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:12:07.521 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T06:12:07.522 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T06:12:07.522 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:12:07.522 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:12:08.206 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:12:09.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:12:09.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:12:09.208 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T06:12:09.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T06:12:09.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:12:09.209 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:12:10.009 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:12:11.010 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:12:11.011 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:12:11.011 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T06:12:11.011 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T06:12:11.011 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:12:11.012 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:12:11.716 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:12:12.717 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:12:12.717 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:12:12.718 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T06:12:12.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T06:12:12.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:12:12.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:12:13.491 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:12:14.493 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:12:14.493 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:12:14.493 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T06:12:14.494 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T06:12:14.494 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:12:14.494 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:12:15.108 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:12:16.109 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:12:16.109 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:12:16.110 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T06:12:16.110 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T06:12:16.110 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:12:16.110 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:12:16.934 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:12:17.936 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:12:17.936 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:12:17.936 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T06:12:17.937 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T06:12:17.937 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:12:17.937 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:12:18.399 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:12:19.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:12:19.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:12:19.401 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T06:12:19.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T06:12:19.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:12:19.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:12:19.824 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2277569357,v1:127.0.0.1:6803/2277569357] [v2:127.0.0.1:6804/2277569357,v1:127.0.0.1:6805/2277569357] exists,up 67307176-bf4f-4fd6-bac2-9f1077a7409f 2024-09-16T06:12:19.828 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:12:19.828 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:12:19.828 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:12:19.829 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-16T06:12:19.829 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:120: TEST_snaptrim_stats_multiple_snaps: run_osd td/test-snaptrim-stats 1 --osd_pool_default_pg_autoscale_mode=off 2024-09-16T06:12:19.829 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-09-16T06:12:19.829 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T06:12:19.829 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T06:12:19.829 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T06:12:19.829 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/1 2024-09-16T06:12:19.829 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9b40f4bd-1dcc-4b6e-878d-4e61738f3b4d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-09-16T06:12:19.829 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:12:19.830 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:12:19.830 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:12:19.830 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/1' 2024-09-16T06:12:19.830 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/1/journal' 2024-09-16T06:12:19.830 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T06:12:19.830 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T06:12:19.830 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-09-16T06:12:19.830 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T06:12:19.830 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:12:19.830 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:12:19.830 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:12:19.830 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:12:19.830 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176784 2024-09-16T06:12:19.830 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176784/$cluster-$name.asok' 2024-09-16T06:12:19.831 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.176784/$cluster-$name.asok' 2024-09-16T06:12:19.831 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:12:19.831 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T06:12:19.831 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T06:12:19.831 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-09-16T06:12:19.831 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-09-16T06:12:19.831 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:12:19.831 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:12:19.831 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:12:19.831 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:12:19.831 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T06:12:19.831 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-09-16T06:12:19.831 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/1 2024-09-16T06:12:19.832 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T06:12:19.832 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f32cd19b-43ee-4bbf-b1a3-cf29dffa701c 2024-09-16T06:12:19.832 INFO:tasks.workunit.client.0.smithi178.stdout:add osd1 f32cd19b-43ee-4bbf-b1a3-cf29dffa701c 2024-09-16T06:12:19.832 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f32cd19b-43ee-4bbf-b1a3-cf29dffa701c' 2024-09-16T06:12:19.832 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T06:12:19.845 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBDzOdmAj9WMhAATX3K3Lep+TpclQtzhTxnqA== 2024-09-16T06:12:19.845 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBDzOdmAj9WMhAATX3K3Lep+TpclQtzhTxnqA=="}' 2024-09-16T06:12:19.845 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f32cd19b-43ee-4bbf-b1a3-cf29dffa701c -i td/test-snaptrim-stats/1/new.json 2024-09-16T06:12:20.271 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:12:20.282 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/1/new.json 2024-09-16T06:12:20.283 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=9b40f4bd-1dcc-4b6e-878d-4e61738f3b4d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/1 --osd-journal=td/test-snaptrim-stats/1/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.176784/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQBDzOdmAj9WMhAATX3K3Lep+TpclQtzhTxnqA== --osd-uuid f32cd19b-43ee-4bbf-b1a3-cf29dffa701c 2024-09-16T06:12:20.312 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:12:20.311+0000 7f7521751640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:12:20.319 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:12:20.319+0000 7f7521751640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:12:20.320 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:12:20.320+0000 7f7521751640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:12:20.321 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:12:20.320+0000 7f7521751640 -1 bdev(0x556f555a5c00 td/test-snaptrim-stats/1/block) open stat got: (1) Operation not permitted 2024-09-16T06:12:20.321 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:12:20.320+0000 7f7521751640 -1 bluestore(td/test-snaptrim-stats/1) _read_fsid unparsable uuid 2024-09-16T06:12:26.370 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/1/keyring 2024-09-16T06:12:26.371 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T06:12:26.371 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd1 key to auth repository 2024-09-16T06:12:26.372 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T06:12:26.372 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T06:12:26.879 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.1 2024-09-16T06:12:26.879 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T06:12:26.879 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=9b40f4bd-1dcc-4b6e-878d-4e61738f3b4d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/1 --osd-journal=td/test-snaptrim-stats/1/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.176784/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-09-16T06:12:26.879 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T06:12:26.880 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T06:12:26.880 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T06:12:26.905 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:12:26.904+0000 7fa49820d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:12:26.925 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:12:26.925+0000 7fa49820d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:12:26.926 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:12:26.926+0000 7fa49820d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:12:27.305 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T06:12:27.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:12:27.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T06:12:27.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:12:27.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:12:27.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:12:27.306 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:12:27.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:12:27.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:12:27.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:12:27.714 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:12:28.524 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:12:28.524+0000 7fa49820d640 -1 Falling back to public interface 2024-09-16T06:12:28.716 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:12:28.716 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:12:28.716 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:12:28.717 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T06:12:28.717 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:12:28.717 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:12:29.157 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:12:30.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:12:30.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:12:30.158 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:12:30.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T06:12:30.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:12:30.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:12:30.575 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:12:31.297 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:12:31.296+0000 7fa49820d640 -1 osd.1 0 log_to_monitors true 2024-09-16T06:12:31.577 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:12:31.577 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:12:31.577 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T06:12:31.578 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T06:12:31.578 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:12:31.578 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:12:32.017 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:12:33.018 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:12:33.018 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:12:33.019 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T06:12:33.019 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T06:12:33.019 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:12:33.019 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:12:33.432 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:12:34.433 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:12:34.433 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:12:34.434 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T06:12:34.434 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T06:12:34.434 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:12:34.434 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:12:34.830 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:12:35.831 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:12:35.832 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:12:35.832 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T06:12:35.832 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T06:12:35.832 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:12:35.833 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:12:36.251 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:12:37.253 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:12:37.253 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:12:37.253 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T06:12:37.254 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T06:12:37.254 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:12:37.254 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:12:37.672 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:12:38.673 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:12:38.674 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:12:38.674 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T06:12:38.674 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T06:12:38.674 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:12:38.674 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:12:39.241 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:12:40.242 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:12:40.243 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:12:40.243 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T06:12:40.243 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T06:12:40.243 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:12:40.243 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:12:40.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:12:41.625 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:12:41.625 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:12:41.626 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T06:12:41.626 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T06:12:41.626 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:12:41.626 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:12:42.174 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:12:43.175 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:12:43.176 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T06:12:43.176 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:12:43.177 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T06:12:43.177 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:12:43.177 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:12:44.070 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:12:45.071 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:12:45.072 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:12:45.072 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T06:12:45.072 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T06:12:45.072 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:12:45.072 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:12:45.653 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:12:46.654 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:12:46.654 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:12:46.655 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T06:12:46.655 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T06:12:46.655 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:12:46.655 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:12:47.438 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:12:48.439 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:12:48.439 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:12:48.439 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T06:12:48.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T06:12:48.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:12:48.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:12:49.125 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:12:50.127 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:12:50.127 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T06:12:50.128 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:12:50.128 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T06:12:50.128 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:12:50.128 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:12:50.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:12:51.897 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:12:51.897 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:12:51.897 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T06:12:51.897 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T06:12:51.898 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:12:51.898 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:12:52.867 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:12:53.869 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:12:53.869 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:12:53.869 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T06:12:53.869 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T06:12:53.870 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:12:53.870 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:12:54.979 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:12:55.981 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:12:55.981 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:12:55.981 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T06:12:55.982 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T06:12:55.982 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:12:55.982 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:12:56.347 INFO:tasks.workunit.client.0.smithi178.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/958084043,v1:127.0.0.1:6811/958084043] [v2:127.0.0.1:6812/958084043,v1:127.0.0.1:6813/958084043] exists,up f32cd19b-43ee-4bbf-b1a3-cf29dffa701c 2024-09-16T06:12:56.351 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:12:56.351 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:12:56.351 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:12:56.351 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-16T06:12:56.351 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:120: TEST_snaptrim_stats_multiple_snaps: run_osd td/test-snaptrim-stats 2 --osd_pool_default_pg_autoscale_mode=off 2024-09-16T06:12:56.351 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-09-16T06:12:56.351 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T06:12:56.351 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T06:12:56.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T06:12:56.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/2 2024-09-16T06:12:56.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9b40f4bd-1dcc-4b6e-878d-4e61738f3b4d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-09-16T06:12:56.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:12:56.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:12:56.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:12:56.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/2' 2024-09-16T06:12:56.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/2/journal' 2024-09-16T06:12:56.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T06:12:56.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T06:12:56.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-09-16T06:12:56.352 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T06:12:56.352 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:12:56.353 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:12:56.353 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:12:56.353 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:12:56.353 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176784 2024-09-16T06:12:56.353 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176784/$cluster-$name.asok' 2024-09-16T06:12:56.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.176784/$cluster-$name.asok' 2024-09-16T06:12:56.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:12:56.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T06:12:56.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T06:12:56.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-09-16T06:12:56.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-09-16T06:12:56.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:12:56.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:12:56.354 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:12:56.354 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:12:56.354 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T06:12:56.354 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-09-16T06:12:56.354 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/2 2024-09-16T06:12:56.354 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T06:12:56.354 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5d50fdcd-c637-42d6-ad70-357594fea4b0 2024-09-16T06:12:56.354 INFO:tasks.workunit.client.0.smithi178.stdout:add osd2 5d50fdcd-c637-42d6-ad70-357594fea4b0 2024-09-16T06:12:56.355 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 5d50fdcd-c637-42d6-ad70-357594fea4b0' 2024-09-16T06:12:56.355 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T06:12:56.368 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBozOdmpITlFRAADjt9prWbtqZa+uLPrvchwQ== 2024-09-16T06:12:56.368 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBozOdmpITlFRAADjt9prWbtqZa+uLPrvchwQ=="}' 2024-09-16T06:12:56.368 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5d50fdcd-c637-42d6-ad70-357594fea4b0 -i td/test-snaptrim-stats/2/new.json 2024-09-16T06:12:56.924 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:12:56.935 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/2/new.json 2024-09-16T06:12:56.936 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=9b40f4bd-1dcc-4b6e-878d-4e61738f3b4d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/2 --osd-journal=td/test-snaptrim-stats/2/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.176784/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQBozOdmpITlFRAADjt9prWbtqZa+uLPrvchwQ== --osd-uuid 5d50fdcd-c637-42d6-ad70-357594fea4b0 2024-09-16T06:12:56.963 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:12:56.963+0000 7fa2ec21a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:12:56.970 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:12:56.970+0000 7fa2ec21a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:12:56.972 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:12:56.971+0000 7fa2ec21a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:12:56.972 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:12:56.972+0000 7fa2ec21a640 -1 bdev(0x55695cb91c00 td/test-snaptrim-stats/2/block) open stat got: (1) Operation not permitted 2024-09-16T06:12:56.972 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:12:56.972+0000 7fa2ec21a640 -1 bluestore(td/test-snaptrim-stats/2) _read_fsid unparsable uuid 2024-09-16T06:13:02.690 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/2/keyring 2024-09-16T06:13:02.691 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T06:13:02.691 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd2 key to auth repository 2024-09-16T06:13:02.692 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T06:13:02.692 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T06:13:03.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T06:13:03.207 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.2 2024-09-16T06:13:03.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=9b40f4bd-1dcc-4b6e-878d-4e61738f3b4d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/2 --osd-journal=td/test-snaptrim-stats/2/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.176784/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-09-16T06:13:03.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T06:13:03.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T06:13:03.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T06:13:03.233 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:13:03.232+0000 7fcf95fdd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:13:03.254 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:13:03.254+0000 7fcf95fdd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:13:03.255 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:13:03.255+0000 7fcf95fdd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:13:03.660 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T06:13:03.660 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:13:03.660 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T06:13:03.660 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:13:03.660 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:13:03.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:13:03.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:13:03.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:13:03.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:13:03.661 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:13:04.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:13:04.848 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:13:04.848+0000 7fcf95fdd640 -1 Falling back to public interface 2024-09-16T06:13:05.032 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:13:05.032 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:13:05.032 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T06:13:05.032 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:13:05.033 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:13:05.033 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:13:05.467 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:13:06.468 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:13:06.468 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:13:06.468 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:13:06.469 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T06:13:06.469 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:13:06.469 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:13:06.880 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:13:07.786 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:13:07.785+0000 7fcf95fdd640 -1 osd.2 0 log_to_monitors true 2024-09-16T06:13:07.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:13:07.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:13:07.882 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T06:13:07.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T06:13:07.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:13:07.883 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:13:08.455 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:13:09.457 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:13:09.457 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:13:09.457 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T06:13:09.458 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T06:13:09.458 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:13:09.458 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:13:09.898 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:13:10.900 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:13:10.900 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:13:10.900 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T06:13:10.900 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T06:13:10.901 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:13:10.901 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:13:11.302 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:13:12.303 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:13:12.303 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:13:12.303 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T06:13:12.304 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T06:13:12.304 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:13:12.304 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:13:12.760 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:13:13.761 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:13:13.761 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:13:13.761 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T06:13:13.762 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T06:13:13.762 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:13:13.762 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:13:14.192 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:13:15.193 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:13:15.193 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:13:15.194 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T06:13:15.194 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T06:13:15.194 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:13:15.194 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:13:15.633 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:13:16.634 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:13:16.634 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:13:16.634 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T06:13:16.635 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T06:13:16.635 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:13:16.635 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:13:17.015 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:13:18.016 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:13:18.016 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:13:18.017 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T06:13:18.017 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T06:13:18.017 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:13:18.017 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:13:18.394 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:13:19.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:13:19.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:13:19.396 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T06:13:19.396 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T06:13:19.396 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:13:19.396 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:13:20.206 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:13:21.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:13:21.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:13:21.208 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T06:13:21.209 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T06:13:21.209 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:13:21.209 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:13:22.176 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:13:23.177 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:13:23.177 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:13:23.178 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T06:13:23.178 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T06:13:23.178 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:13:23.178 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:13:24.162 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:13:25.163 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:13:25.164 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:13:25.164 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T06:13:25.164 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T06:13:25.164 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:13:25.165 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:13:26.029 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:13:27.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:13:27.030 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:13:27.031 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T06:13:27.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T06:13:27.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:13:27.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:13:27.824 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:13:28.826 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:13:28.826 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:13:28.826 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T06:13:28.826 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T06:13:28.827 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:13:28.827 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:13:29.800 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:13:30.801 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:13:30.801 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:13:30.802 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T06:13:30.802 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T06:13:30.802 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:13:30.802 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:13:31.242 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:13:32.243 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:13:32.243 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:13:32.244 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T06:13:32.244 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T06:13:32.244 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:13:32.244 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:13:32.749 INFO:tasks.workunit.client.0.smithi178.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/215301645,v1:127.0.0.1:6819/215301645] [v2:127.0.0.1:6820/215301645,v1:127.0.0.1:6821/215301645] exists,up 5d50fdcd-c637-42d6-ad70-357594fea4b0 2024-09-16T06:13:32.750 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:13:32.750 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:13:32.750 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:13:32.750 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:124: TEST_snaptrim_stats_multiple_snaps: ceph osd set noscrub 2024-09-16T06:13:33.331 INFO:tasks.workunit.client.0.smithi178.stderr:noscrub is set 2024-09-16T06:13:33.341 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:125: TEST_snaptrim_stats_multiple_snaps: ceph osd set nodeep-scrub 2024-09-16T06:13:34.026 INFO:tasks.workunit.client.0.smithi178.stderr:nodeep-scrub is set 2024-09-16T06:13:34.038 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:128: TEST_snaptrim_stats_multiple_snaps: create_pool test 8 8 2024-09-16T06:13:34.038 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 8 8 2024-09-16T06:13:34.589 INFO:tasks.workunit.client.0.smithi178.stderr:pool 'test' already exists 2024-09-16T06:13:34.600 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T06:13:35.602 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:129: TEST_snaptrim_stats_multiple_snaps: wait_for_clean 2024-09-16T06:13:35.602 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T06:13:35.602 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T06:13:35.602 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T06:13:35.602 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T06:13:35.603 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T06:13:35.603 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T06:13:35.603 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T06:13:35.603 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T06:13:35.603 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T06:13:35.694 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T06:13:35.694 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T06:13:35.694 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T06:13:35.694 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T06:13:35.694 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-16T06:13:35.695 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-16T06:13:36.377 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-16T06:13:36.377 INFO:tasks.workunit.client.0.smithi178.stderr:1 2024-09-16T06:13:36.377 INFO:tasks.workunit.client.0.smithi178.stderr:2' 2024-09-16T06:13:36.377 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-16T06:13:36.377 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T06:13:36.377 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T06:13:36.510 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836501 2024-09-16T06:13:36.510 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836501 2024-09-16T06:13:36.510 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501' 2024-09-16T06:13:36.511 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T06:13:36.511 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T06:13:36.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672973 2024-09-16T06:13:36.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672973 2024-09-16T06:13:36.622 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672973' 2024-09-16T06:13:36.622 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T06:13:36.622 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T06:13:36.745 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509446 2024-09-16T06:13:36.746 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509446 2024-09-16T06:13:36.746 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672973 2-64424509446' 2024-09-16T06:13:36.746 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T06:13:36.746 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836501 2024-09-16T06:13:36.746 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T06:13:36.747 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-16T06:13:36.747 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836501 2024-09-16T06:13:36.747 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T06:13:36.748 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836501 2024-09-16T06:13:36.748 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-09-16T06:13:36.749 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.0 seq 21474836501 2024-09-16T06:13:36.749 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T06:13:37.117 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836501 2024-09-16T06:13:37.117 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-16T06:13:38.118 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T06:13:38.118 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T06:13:38.488 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836501 2024-09-16T06:13:38.488 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T06:13:38.489 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672973 2024-09-16T06:13:38.489 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T06:13:38.489 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-09-16T06:13:38.490 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672973 2024-09-16T06:13:38.490 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T06:13:38.491 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672973 2024-09-16T06:13:38.491 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-09-16T06:13:38.491 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.1 seq 42949672973 2024-09-16T06:13:38.491 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T06:13:38.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672973 -lt 42949672973 2024-09-16T06:13:38.859 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T06:13:38.859 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509446 2024-09-16T06:13:38.859 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T06:13:38.860 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-09-16T06:13:38.860 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509446 2024-09-16T06:13:38.860 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T06:13:38.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509446 2024-09-16T06:13:38.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-09-16T06:13:38.862 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.2 seq 64424509446 2024-09-16T06:13:38.862 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T06:13:39.279 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509447 -lt 64424509446 2024-09-16T06:13:39.279 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T06:13:39.279 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T06:13:39.279 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T06:13:39.780 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-09-16T06:13:39.780 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T06:13:39.781 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T06:13:39.781 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T06:13:39.781 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T06:13:39.781 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T06:13:39.781 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T06:13:39.781 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T06:13:40.143 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-16T06:13:40.143 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T06:13:40.143 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T06:13:40.143 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T06:13:40.638 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-09-16T06:13:40.638 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T06:13:40.638 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T06:13:40.639 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:130: TEST_snaptrim_stats_multiple_snaps: ceph osd dump 2024-09-16T06:13:40.639 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:130: TEST_snaptrim_stats_multiple_snaps: grep '^pool.*['\'']test['\'']' 2024-09-16T06:13:40.639 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:130: TEST_snaptrim_stats_multiple_snaps: awk '{ print $2 }' 2024-09-16T06:13:41.011 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:130: TEST_snaptrim_stats_multiple_snaps: poolid=1 2024-09-16T06:13:41.011 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:133: TEST_snaptrim_stats_multiple_snaps: local TESTDATA=testdata.0 2024-09-16T06:13:41.011 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:134: TEST_snaptrim_stats_multiple_snaps: dd if=/dev/urandom of=testdata.0 bs=4096 count=1 2024-09-16T06:13:41.012 INFO:tasks.workunit.client.0.smithi178.stderr:1+0 records in 2024-09-16T06:13:41.012 INFO:tasks.workunit.client.0.smithi178.stderr:1+0 records out 2024-09-16T06:13:41.012 INFO:tasks.workunit.client.0.smithi178.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 6.0763e-05 s, 67.4 MB/s 2024-09-16T06:13:41.012 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: seq 1 10 2024-09-16T06:13:41.012 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T06:13:41.013 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj1 testdata.0 2024-09-16T06:13:41.198 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T06:13:41.198 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj2 testdata.0 2024-09-16T06:13:41.366 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T06:13:41.366 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj3 testdata.0 2024-09-16T06:13:41.525 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T06:13:41.525 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj4 testdata.0 2024-09-16T06:13:41.710 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T06:13:41.710 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj5 testdata.0 2024-09-16T06:13:41.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T06:13:41.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj6 testdata.0 2024-09-16T06:13:42.020 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T06:13:42.020 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj7 testdata.0 2024-09-16T06:13:42.154 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T06:13:42.154 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj8 testdata.0 2024-09-16T06:13:42.296 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T06:13:42.297 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj9 testdata.0 2024-09-16T06:13:42.431 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T06:13:42.431 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj10 testdata.0 2024-09-16T06:13:42.581 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:139: TEST_snaptrim_stats_multiple_snaps: rm -f testdata.0 2024-09-16T06:13:42.582 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:142: TEST_snaptrim_stats_multiple_snaps: NUMSNAPS=2 2024-09-16T06:13:42.582 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:143: TEST_snaptrim_stats_multiple_snaps: seq 1 2 2024-09-16T06:13:42.583 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:143: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $NUMSNAPS` 2024-09-16T06:13:42.584 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:145: TEST_snaptrim_stats_multiple_snaps: rados -p test mksnap snap1 2024-09-16T06:13:42.731 INFO:tasks.workunit.client.0.smithi178.stdout:created pool test snap snap1 2024-09-16T06:13:42.733 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:146: TEST_snaptrim_stats_multiple_snaps: TESTDATA=testdata.1 2024-09-16T06:13:42.733 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:147: TEST_snaptrim_stats_multiple_snaps: dd if=/dev/urandom of=testdata.1 bs=4096 count=1 2024-09-16T06:13:42.734 INFO:tasks.workunit.client.0.smithi178.stderr:1+0 records in 2024-09-16T06:13:42.734 INFO:tasks.workunit.client.0.smithi178.stderr:1+0 records out 2024-09-16T06:13:42.734 INFO:tasks.workunit.client.0.smithi178.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 5.247e-05 s, 78.1 MB/s 2024-09-16T06:13:42.734 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: seq 1 10 2024-09-16T06:13:42.735 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T06:13:42.735 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj1 testdata.1 2024-09-16T06:13:43.109 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T06:13:43.109 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj2 testdata.1 2024-09-16T06:13:43.276 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T06:13:43.276 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj3 testdata.1 2024-09-16T06:13:43.461 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T06:13:43.461 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj4 testdata.1 2024-09-16T06:13:43.603 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T06:13:43.603 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj5 testdata.1 2024-09-16T06:13:43.737 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T06:13:43.738 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj6 testdata.1 2024-09-16T06:13:43.880 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T06:13:43.880 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj7 testdata.1 2024-09-16T06:13:44.014 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T06:13:44.014 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj8 testdata.1 2024-09-16T06:13:44.164 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T06:13:44.165 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj9 testdata.1 2024-09-16T06:13:44.315 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T06:13:44.315 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj10 testdata.1 2024-09-16T06:13:44.491 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:152: TEST_snaptrim_stats_multiple_snaps: rm -f testdata.1 2024-09-16T06:13:44.492 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:143: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $NUMSNAPS` 2024-09-16T06:13:44.492 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:145: TEST_snaptrim_stats_multiple_snaps: rados -p test mksnap snap2 2024-09-16T06:13:44.641 INFO:tasks.workunit.client.0.smithi178.stdout:created pool test snap snap2 2024-09-16T06:13:44.643 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:146: TEST_snaptrim_stats_multiple_snaps: TESTDATA=testdata.2 2024-09-16T06:13:44.643 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:147: TEST_snaptrim_stats_multiple_snaps: dd if=/dev/urandom of=testdata.2 bs=4096 count=1 2024-09-16T06:13:44.643 INFO:tasks.workunit.client.0.smithi178.stderr:1+0 records in 2024-09-16T06:13:44.643 INFO:tasks.workunit.client.0.smithi178.stderr:1+0 records out 2024-09-16T06:13:44.644 INFO:tasks.workunit.client.0.smithi178.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 4.95e-05 s, 82.7 MB/s 2024-09-16T06:13:44.644 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: seq 1 10 2024-09-16T06:13:44.645 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T06:13:44.645 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj1 testdata.2 2024-09-16T06:13:44.977 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T06:13:44.977 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj2 testdata.2 2024-09-16T06:13:45.125 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T06:13:45.126 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj3 testdata.2 2024-09-16T06:13:45.270 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T06:13:45.270 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj4 testdata.2 2024-09-16T06:13:45.455 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T06:13:45.455 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj5 testdata.2 2024-09-16T06:13:45.606 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T06:13:45.606 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj6 testdata.2 2024-09-16T06:13:45.756 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T06:13:45.757 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj7 testdata.2 2024-09-16T06:13:45.932 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T06:13:45.932 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj8 testdata.2 2024-09-16T06:13:46.083 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T06:13:46.083 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj9 testdata.2 2024-09-16T06:13:46.226 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T06:13:46.226 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj10 testdata.2 2024-09-16T06:13:46.376 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:152: TEST_snaptrim_stats_multiple_snaps: rm -f testdata.2 2024-09-16T06:13:46.376 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:156: TEST_snaptrim_stats_multiple_snaps: local total_objects_trimmed=0 2024-09-16T06:13:46.377 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:157: TEST_snaptrim_stats_multiple_snaps: seq 1 2 2024-09-16T06:13:46.378 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:157: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $NUMSNAPS` 2024-09-16T06:13:46.378 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:159: TEST_snaptrim_stats_multiple_snaps: rados -p test rmsnap snap1 2024-09-16T06:13:46.551 INFO:tasks.workunit.client.0.smithi178.stdout:removed pool test snap snap1 2024-09-16T06:13:46.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:162: TEST_snaptrim_stats_multiple_snaps: wait_for_clean 2024-09-16T06:13:46.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T06:13:46.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T06:13:46.554 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T06:13:46.554 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T06:13:46.555 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T06:13:46.555 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T06:13:46.555 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T06:13:46.555 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T06:13:46.555 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T06:13:46.640 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T06:13:46.640 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T06:13:46.641 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T06:13:46.641 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T06:13:46.641 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-16T06:13:46.641 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-16T06:13:47.132 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-16T06:13:47.133 INFO:tasks.workunit.client.0.smithi178.stderr:1 2024-09-16T06:13:47.133 INFO:tasks.workunit.client.0.smithi178.stderr:2' 2024-09-16T06:13:47.133 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-16T06:13:47.133 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T06:13:47.133 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T06:13:47.247 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836505 2024-09-16T06:13:47.247 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836505 2024-09-16T06:13:47.247 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505' 2024-09-16T06:13:47.247 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T06:13:47.247 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T06:13:47.512 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672977 2024-09-16T06:13:47.512 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672977 2024-09-16T06:13:47.512 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672977' 2024-09-16T06:13:47.512 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T06:13:47.512 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T06:13:47.641 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509451 2024-09-16T06:13:47.641 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509451 2024-09-16T06:13:47.642 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672977 2-64424509451' 2024-09-16T06:13:47.642 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T06:13:47.642 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836505 2024-09-16T06:13:47.642 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T06:13:47.642 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-16T06:13:47.643 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836505 2024-09-16T06:13:47.643 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T06:13:47.644 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836505 2024-09-16T06:13:47.644 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-09-16T06:13:47.644 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.0 seq 21474836505 2024-09-16T06:13:47.644 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T06:13:48.094 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836505 2024-09-16T06:13:48.094 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-09-16T06:13:49.095 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T06:13:49.095 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T06:13:49.483 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836505 2024-09-16T06:13:49.483 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T06:13:49.483 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672977 2024-09-16T06:13:49.483 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T06:13:49.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-09-16T06:13:49.485 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672977 2024-09-16T06:13:49.485 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T06:13:49.486 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672977 2024-09-16T06:13:49.486 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-09-16T06:13:49.486 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.1 seq 42949672977 2024-09-16T06:13:49.486 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T06:13:49.860 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672977 -lt 42949672977 2024-09-16T06:13:49.860 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T06:13:49.860 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509451 2024-09-16T06:13:49.860 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T06:13:49.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-09-16T06:13:49.862 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509451 2024-09-16T06:13:49.862 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T06:13:49.862 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.2 seq 64424509451 2024-09-16T06:13:49.862 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509451 2024-09-16T06:13:49.863 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-09-16T06:13:49.863 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T06:13:50.231 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509451 -lt 64424509451 2024-09-16T06:13:50.232 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T06:13:50.232 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T06:13:50.232 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T06:13:50.731 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-09-16T06:13:50.731 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T06:13:50.731 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T06:13:50.731 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T06:13:50.732 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T06:13:50.732 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T06:13:50.732 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T06:13:50.732 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T06:13:51.133 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-16T06:13:51.134 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T06:13:51.134 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T06:13:51.134 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T06:13:51.634 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-09-16T06:13:51.635 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T06:13:51.635 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T06:13:51.635 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:163: TEST_snaptrim_stats_multiple_snaps: sleep 10 2024-09-16T06:14:01.636 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:164: TEST_snaptrim_stats_multiple_snaps: local objects_trimmed=0 2024-09-16T06:14:01.636 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:165: TEST_snaptrim_stats_multiple_snaps: local snaptrim_duration_total=0.0 2024-09-16T06:14:01.636 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: expr 8 - 1 2024-09-16T06:14:01.638 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: seq 0 7 2024-09-16T06:14:01.639 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T06:14:01.639 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.0 2024-09-16T06:14:01.640 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.0 query 2024-09-16T06:14:01.640 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-16T06:14:01.767 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 0 + 2 2024-09-16T06:14:01.768 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=2 2024-09-16T06:14:01.768 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-16T06:14:01.769 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.0 query 2024-09-16T06:14:01.769 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-16T06:14:01.886 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 0.0 + 0.528189397 2024-09-16T06:14:01.887 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.528189397 2024-09-16T06:14:01.887 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T06:14:01.887 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.1 2024-09-16T06:14:01.887 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.1 query 2024-09-16T06:14:01.887 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-16T06:14:02.002 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 2 + 2 2024-09-16T06:14:02.003 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=4 2024-09-16T06:14:02.003 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-16T06:14:02.004 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.1 query 2024-09-16T06:14:02.004 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-16T06:14:02.119 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .528189397 + 0.488387395 2024-09-16T06:14:02.119 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.016576792 2024-09-16T06:14:02.120 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T06:14:02.120 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.2 2024-09-16T06:14:02.120 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.2 query 2024-09-16T06:14:02.120 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-16T06:14:02.236 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 4 + 0 2024-09-16T06:14:02.236 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=4 2024-09-16T06:14:02.237 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-16T06:14:02.237 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.2 query 2024-09-16T06:14:02.238 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-16T06:14:02.352 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.016576792 + 0.00045737499999999999 2024-09-16T06:14:02.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.01703416699999999999 2024-09-16T06:14:02.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T06:14:02.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.3 2024-09-16T06:14:02.354 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.3 query 2024-09-16T06:14:02.354 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-16T06:14:02.468 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 4 + 2 2024-09-16T06:14:02.470 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=6 2024-09-16T06:14:02.470 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-16T06:14:02.470 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.3 query 2024-09-16T06:14:02.470 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-16T06:14:02.586 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.01703416699999999999 + 0.79827234000000002 2024-09-16T06:14:02.587 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.81530650700000001999 2024-09-16T06:14:02.587 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T06:14:02.587 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.4 2024-09-16T06:14:02.587 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.4 query 2024-09-16T06:14:02.587 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-16T06:14:02.703 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 6 + 1 2024-09-16T06:14:02.704 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=7 2024-09-16T06:14:02.704 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-16T06:14:02.705 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.4 query 2024-09-16T06:14:02.705 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-16T06:14:02.820 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.81530650700000001999 + 0.39436128300000001 2024-09-16T06:14:02.821 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=2.20966779000000002999 2024-09-16T06:14:02.821 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T06:14:02.821 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.5 2024-09-16T06:14:02.822 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.5 query 2024-09-16T06:14:02.822 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-16T06:14:02.937 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 7 + 1 2024-09-16T06:14:02.938 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=8 2024-09-16T06:14:02.938 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-16T06:14:02.939 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.5 query 2024-09-16T06:14:02.939 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-16T06:14:03.055 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 2.20966779000000002999 + 0.488411021 2024-09-16T06:14:03.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=2.69807881100000002999 2024-09-16T06:14:03.056 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T06:14:03.056 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.6 2024-09-16T06:14:03.056 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.6 query 2024-09-16T06:14:03.056 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-16T06:14:03.183 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 8 + 0 2024-09-16T06:14:03.183 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=8 2024-09-16T06:14:03.184 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-16T06:14:03.185 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.6 query 2024-09-16T06:14:03.185 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-16T06:14:03.301 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 2.69807881100000002999 + 0.929988814 2024-09-16T06:14:03.302 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=3.62806762500000002999 2024-09-16T06:14:03.302 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T06:14:03.302 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.7 2024-09-16T06:14:03.302 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.7 query 2024-09-16T06:14:03.302 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-16T06:14:03.419 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 8 + 2 2024-09-16T06:14:03.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=10 2024-09-16T06:14:03.420 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-16T06:14:03.421 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.7 query 2024-09-16T06:14:03.421 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-16T06:14:03.536 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 3.62806762500000002999 + 1.064657167 2024-09-16T06:14:03.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=4.69272479200000002999 2024-09-16T06:14:03.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:174: TEST_snaptrim_stats_multiple_snaps: test 10 -eq 10 2024-09-16T06:14:03.537 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:175: TEST_snaptrim_stats_multiple_snaps: echo '4.69272479200000002999 > 0.0' 2024-09-16T06:14:03.537 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:175: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-16T06:14:03.537 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:14:03.537 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:176: TEST_snaptrim_stats_multiple_snaps: expr 0 + 10 2024-09-16T06:14:03.539 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:176: TEST_snaptrim_stats_multiple_snaps: total_objects_trimmed=10 2024-09-16T06:14:03.539 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:157: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $NUMSNAPS` 2024-09-16T06:14:03.539 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:159: TEST_snaptrim_stats_multiple_snaps: rados -p test rmsnap snap2 2024-09-16T06:14:03.685 INFO:tasks.workunit.client.0.smithi178.stdout:removed pool test snap snap2 2024-09-16T06:14:03.688 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:162: TEST_snaptrim_stats_multiple_snaps: wait_for_clean 2024-09-16T06:14:03.688 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T06:14:03.688 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T06:14:03.688 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T06:14:03.688 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T06:14:03.689 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T06:14:03.689 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T06:14:03.689 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T06:14:03.689 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T06:14:03.689 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T06:14:03.775 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T06:14:03.775 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T06:14:03.775 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T06:14:03.775 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T06:14:03.775 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-09-16T06:14:03.775 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-09-16T06:14:04.157 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-09-16T06:14:04.157 INFO:tasks.workunit.client.0.smithi178.stderr:1 2024-09-16T06:14:04.157 INFO:tasks.workunit.client.0.smithi178.stderr:2' 2024-09-16T06:14:04.157 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-09-16T06:14:04.157 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T06:14:04.157 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T06:14:04.377 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836511 2024-09-16T06:14:04.378 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836511 2024-09-16T06:14:04.378 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511' 2024-09-16T06:14:04.378 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T06:14:04.378 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T06:14:04.709 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672982 2024-09-16T06:14:04.709 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672982 2024-09-16T06:14:04.709 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511 1-42949672982' 2024-09-16T06:14:04.709 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-09-16T06:14:04.710 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T06:14:04.820 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-09-16T06:14:04.820 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-09-16T06:14:04.820 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511 1-42949672982 2-64424509456' 2024-09-16T06:14:04.820 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T06:14:04.820 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836511 2024-09-16T06:14:04.820 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T06:14:04.821 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-09-16T06:14:04.821 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836511 2024-09-16T06:14:04.822 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T06:14:04.822 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836511 2024-09-16T06:14:04.823 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.0 seq 21474836511 2024-09-16T06:14:04.823 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-09-16T06:14:04.823 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T06:14:05.203 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836511 -lt 21474836511 2024-09-16T06:14:05.204 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T06:14:05.204 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672982 2024-09-16T06:14:05.204 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T06:14:05.205 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-09-16T06:14:05.205 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672982 2024-09-16T06:14:05.205 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T06:14:05.206 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672982 2024-09-16T06:14:05.206 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.1 seq 42949672982 2024-09-16T06:14:05.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-09-16T06:14:05.207 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T06:14:05.582 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672982 -lt 42949672982 2024-09-16T06:14:05.582 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-09-16T06:14:05.582 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-09-16T06:14:05.582 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-09-16T06:14:05.583 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-09-16T06:14:05.583 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-09-16T06:14:05.584 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-09-16T06:14:05.585 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-09-16T06:14:05.585 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-09-16T06:14:05.585 INFO:tasks.workunit.client.0.smithi178.stdout:waiting osd.2 seq 64424509456 2024-09-16T06:14:05.585 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T06:14:05.954 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509456 2024-09-16T06:14:05.955 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T06:14:05.955 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T06:14:05.955 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T06:14:06.453 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-09-16T06:14:06.454 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T06:14:06.454 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T06:14:06.454 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T06:14:06.454 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T06:14:06.454 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T06:14:06.454 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T06:14:06.454 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T06:14:06.813 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-16T06:14:06.813 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T06:14:06.813 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T06:14:06.813 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T06:14:07.320 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-09-16T06:14:07.321 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T06:14:07.321 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T06:14:07.321 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:163: TEST_snaptrim_stats_multiple_snaps: sleep 10 2024-09-16T06:14:17.323 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:164: TEST_snaptrim_stats_multiple_snaps: local objects_trimmed=0 2024-09-16T06:14:17.323 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:165: TEST_snaptrim_stats_multiple_snaps: local snaptrim_duration_total=0.0 2024-09-16T06:14:17.323 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: expr 8 - 1 2024-09-16T06:14:17.324 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: seq 0 7 2024-09-16T06:14:17.325 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T06:14:17.325 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.0 2024-09-16T06:14:17.326 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.0 query 2024-09-16T06:14:17.327 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-16T06:14:17.454 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 0 + 2 2024-09-16T06:14:17.455 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=2 2024-09-16T06:14:17.455 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-16T06:14:17.456 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.0 query 2024-09-16T06:14:17.456 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-16T06:14:17.572 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 0.0 + 0.84607091400000001 2024-09-16T06:14:17.573 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.84607091400000001 2024-09-16T06:14:17.573 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T06:14:17.573 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.1 2024-09-16T06:14:17.573 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.1 query 2024-09-16T06:14:17.574 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-16T06:14:17.690 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 2 + 2 2024-09-16T06:14:17.691 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=4 2024-09-16T06:14:17.691 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-16T06:14:17.692 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.1 query 2024-09-16T06:14:17.692 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-16T06:14:17.808 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .84607091400000001 + 0.50471234799999998 2024-09-16T06:14:17.809 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.35078326199999999 2024-09-16T06:14:17.809 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T06:14:17.809 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.2 2024-09-16T06:14:17.810 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.2 query 2024-09-16T06:14:17.810 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-16T06:14:17.925 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 4 + 0 2024-09-16T06:14:17.926 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=4 2024-09-16T06:14:17.927 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-16T06:14:17.927 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.2 query 2024-09-16T06:14:17.927 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-16T06:14:18.043 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.35078326199999999 + 0.00026477 2024-09-16T06:14:18.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.35104803199999999 2024-09-16T06:14:18.043 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T06:14:18.044 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.3 2024-09-16T06:14:18.044 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.3 query 2024-09-16T06:14:18.044 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-16T06:14:18.160 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 4 + 2 2024-09-16T06:14:18.161 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=6 2024-09-16T06:14:18.161 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-16T06:14:18.162 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.3 query 2024-09-16T06:14:18.162 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-16T06:14:18.277 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.35104803199999999 + 0.335872801 2024-09-16T06:14:18.278 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.68692083299999999 2024-09-16T06:14:18.278 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T06:14:18.278 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.4 2024-09-16T06:14:18.279 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.4 query 2024-09-16T06:14:18.279 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-16T06:14:18.395 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 6 + 1 2024-09-16T06:14:18.396 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=7 2024-09-16T06:14:18.396 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-16T06:14:18.397 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.4 query 2024-09-16T06:14:18.397 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-16T06:14:18.511 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.68692083299999999 + 0.20146430100000001 2024-09-16T06:14:18.512 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.88838513400000000 2024-09-16T06:14:18.512 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T06:14:18.512 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.5 2024-09-16T06:14:18.513 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.5 query 2024-09-16T06:14:18.513 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-16T06:14:18.629 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 7 + 1 2024-09-16T06:14:18.630 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=8 2024-09-16T06:14:18.631 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-16T06:14:18.631 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.5 query 2024-09-16T06:14:18.631 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-16T06:14:18.748 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.88838513400000000 + 0.21817967399999999 2024-09-16T06:14:18.748 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=2.10656480799999999 2024-09-16T06:14:18.749 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T06:14:18.749 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.6 2024-09-16T06:14:18.749 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.6 query 2024-09-16T06:14:18.749 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-16T06:14:18.864 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 8 + 0 2024-09-16T06:14:18.865 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=8 2024-09-16T06:14:18.865 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-16T06:14:18.866 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.6 query 2024-09-16T06:14:18.866 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-16T06:14:18.982 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 2.10656480799999999 + 0.20168297199999999 2024-09-16T06:14:18.983 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=2.30824777999999998 2024-09-16T06:14:18.983 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T06:14:18.983 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.7 2024-09-16T06:14:18.984 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.7 query 2024-09-16T06:14:18.984 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-16T06:14:19.100 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 8 + 2 2024-09-16T06:14:19.101 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=10 2024-09-16T06:14:19.102 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-16T06:14:19.102 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.7 query 2024-09-16T06:14:19.102 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-16T06:14:19.219 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 2.30824777999999998 + 0.61982499800000002 2024-09-16T06:14:19.219 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=2.92807277800000000 2024-09-16T06:14:19.219 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:174: TEST_snaptrim_stats_multiple_snaps: test 10 -eq 10 2024-09-16T06:14:19.219 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:175: TEST_snaptrim_stats_multiple_snaps: echo '2.92807277800000000 > 0.0' 2024-09-16T06:14:19.220 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:175: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-16T06:14:19.220 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:14:19.220 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:176: TEST_snaptrim_stats_multiple_snaps: expr 10 + 10 2024-09-16T06:14:19.222 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:176: TEST_snaptrim_stats_multiple_snaps: total_objects_trimmed=20 2024-09-16T06:14:19.222 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:179: TEST_snaptrim_stats_multiple_snaps: test 20 -eq 20 2024-09-16T06:14:19.222 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:181: TEST_snaptrim_stats_multiple_snaps: teardown td/test-snaptrim-stats 2024-09-16T06:14:19.222 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-09-16T06:14:19.222 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T06:14:19.222 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-09-16T06:14:19.223 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:14:19.223 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:14:19.223 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:14:19.223 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:14:19.223 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:14:19.329 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:14:19.329 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:14:19.330 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:14:19.330 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:14:19.331 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:14:19.331 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:14:19.331 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:14:19.332 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:14:19.332 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:14:19.333 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:14:19.333 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:14:19.333 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:14:19.334 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T06:14:19.334 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-09-16T06:14:19.366 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:14:19.366 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:14:19.367 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176784 2024-09-16T06:14:19.367 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.176784 2024-09-16T06:14:19.368 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:14:19.368 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:14:19.368 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:34: run: teardown td/test-snaptrim-stats 2024-09-16T06:14:19.368 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-09-16T06:14:19.368 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T06:14:19.368 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-09-16T06:14:19.368 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:14:19.368 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:14:19.369 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:14:19.369 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:14:19.369 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:14:19.369 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:14:19.370 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:14:19.370 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:14:19.371 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:14:19.372 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:14:19.372 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:14:19.372 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:14:19.373 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:14:19.373 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:14:19.373 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:14:19.373 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:14:19.374 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:14:19.375 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T06:14:19.375 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-09-16T06:14:19.376 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:14:19.376 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:14:19.376 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176784 2024-09-16T06:14:19.376 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.176784 2024-09-16T06:14:19.377 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:14:19.378 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:14:19.378 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-09-16T06:14:19.378 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/test-snaptrim-stats 0 2024-09-16T06:14:19.378 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-09-16T06:14:19.378 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-16T06:14:19.378 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-09-16T06:14:19.378 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:14:19.378 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:14:19.378 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:14:19.378 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:14:19.378 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:14:19.379 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:14:19.379 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:14:19.380 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:14:19.380 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:14:19.381 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:14:19.381 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:14:19.381 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:14:19.382 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:14:19.382 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:14:19.383 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:14:19.383 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:14:19.383 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:14:19.384 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-16T06:14:19.384 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-09-16T06:14:19.385 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:14:19.385 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:14:19.385 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176784 2024-09-16T06:14:19.386 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.176784 2024-09-16T06:14:19.387 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:14:19.387 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:14:19.387 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-09-16T06:14:19.388 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T06:14:19.388 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T06:14:19.413 INFO:tasks.workunit:Running workunit misc/ver-health.sh... 2024-09-16T06:14:19.413 DEBUG:teuthology.orchestra.run.smithi178:workunit test misc/ver-health.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ad35db06e8924912f01e32b35576297c0f1352b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh 2024-09-16T06:14:19.475 INFO:tasks.workunit.client.0.smithi178.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-16T06:14:19.481 INFO:tasks.workunit.client.0.smithi178.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-16T06:14:19.481 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-16T06:14:19.481 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-16T06:14:19.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-09-16T06:14:19.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-09-16T06:14:19.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-09-16T06:14:19.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-09-16T06:14:19.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-09-16T06:14:19.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-09-16T06:14:19.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/ver-health 2024-09-16T06:14:19.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:20: run: local dir=td/ver-health 2024-09-16T06:14:19.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:21: run: shift 2024-09-16T06:14:19.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:23: run: export CEPH_MON_A=127.0.0.1:7165 2024-09-16T06:14:19.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:23: run: CEPH_MON_A=127.0.0.1:7165 2024-09-16T06:14:19.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:24: run: export CEPH_MON_B=127.0.0.1:7166 2024-09-16T06:14:19.483 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:24: run: CEPH_MON_B=127.0.0.1:7166 2024-09-16T06:14:19.483 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:25: run: export CEPH_ARGS 2024-09-16T06:14:19.483 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:26: run: uuidgen 2024-09-16T06:14:19.483 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:26: run: CEPH_ARGS+='--fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none ' 2024-09-16T06:14:19.483 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:27: run: CEPH_ARGS+='--mon-host= ' 2024-09-16T06:14:19.483 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:28: run: CEPH_ARGS+='--mon_health_to_clog_tick_interval=1.0 ' 2024-09-16T06:14:19.483 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:29: run: export 'ORIG_CEPH_ARGS=--fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 ' 2024-09-16T06:14:19.483 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:29: run: ORIG_CEPH_ARGS='--fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 ' 2024-09-16T06:14:19.483 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:31: run: set 2024-09-16T06:14:19.483 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-16T06:14:19.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:31: run: local 'funcs=TEST_check_version_health_1 2024-09-16T06:14:19.485 INFO:tasks.workunit.client.0.smithi178.stderr:TEST_check_version_health_2 2024-09-16T06:14:19.485 INFO:tasks.workunit.client.0.smithi178.stderr:TEST_check_version_health_3' 2024-09-16T06:14:19.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:32: run: for func in $funcs 2024-09-16T06:14:19.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:33: run: setup td/ver-health 2024-09-16T06:14:19.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2024-09-16T06:14:19.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2024-09-16T06:14:19.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-16T06:14:19.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T06:14:19.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-16T06:14:19.485 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:14:19.485 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:14:19.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:14:19.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:14:19.486 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:14:19.486 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:14:19.486 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:14:19.486 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:14:19.486 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:14:19.487 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:14:19.487 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:14:19.487 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:14:19.488 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:14:19.488 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:14:19.488 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:14:19.488 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:14:19.489 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:14:19.490 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T06:14:19.490 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-16T06:14:19.491 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:14:19.491 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:14:19.491 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:14:19.491 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.192403 2024-09-16T06:14:19.492 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:14:19.492 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:14:19.492 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2024-09-16T06:14:19.493 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T06:14:19.493 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:14:19.493 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:14:19.493 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.192403 2024-09-16T06:14:19.496 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T06:14:19.496 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-16T06:14:19.496 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-16T06:14:19.496 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T06:14:19.496 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2024-09-16T06:14:19.496 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:34: run: TEST_check_version_health_1 td/ver-health 2024-09-16T06:14:19.496 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:68: TEST_check_version_health_1: local dir=td/ver-health 2024-09-16T06:14:19.496 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:71: TEST_check_version_health_1: CEPH_ARGS='--fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T06:14:19.496 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:73: TEST_check_version_health_1: setup td/ver-health 2024-09-16T06:14:19.496 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2024-09-16T06:14:19.497 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2024-09-16T06:14:19.497 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-16T06:14:19.497 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T06:14:19.497 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-16T06:14:19.497 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:14:19.497 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:14:19.497 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:14:19.497 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:14:19.497 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:14:19.498 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:14:19.498 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:14:19.498 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:14:19.498 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:14:19.499 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:14:19.499 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:14:19.499 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:14:19.500 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:14:19.500 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:14:19.500 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:14:19.500 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:14:19.500 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:14:19.501 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T06:14:19.502 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-16T06:14:19.503 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:14:19.503 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:14:19.503 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:14:19.503 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.192403 2024-09-16T06:14:19.504 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:14:19.504 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:14:19.504 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2024-09-16T06:14:19.505 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T06:14:19.505 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:14:19.505 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:14:19.505 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.192403 2024-09-16T06:14:19.507 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T06:14:19.507 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T06:14:19.507 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T06:14:19.507 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2024-09-16T06:14:19.507 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:76: TEST_check_version_health_1: run_mon td/ver-health a --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-09-16T06:14:19.507 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-09-16T06:14:19.507 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T06:14:19.507 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T06:14:19.507 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T06:14:19.508 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/a 2024-09-16T06:14:19.508 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ver-health/a --run-dir=td/ver-health --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-09-16T06:14:19.888 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T06:14:19.888 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:14:19.888 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:14:19.888 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:14:19.888 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:14:19.888 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:14:19.888 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:14:19.888 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/a '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-09-16T06:14:20.080 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T06:14:20.082 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T06:14:20.082 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T06:14:20.082 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T06:14:20.082 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T06:14:20.082 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T06:14:20.082 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T06:14:20.082 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T06:14:20.082 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T06:14:20.082 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:14:20.082 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:14:20.083 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:14:20.083 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.192403/ceph-mon.a.asok 2024-09-16T06:14:20.083 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T06:14:20.083 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.192403/ceph-mon.a.asok config get fsid 2024-09-16T06:14:20.163 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T06:14:20.163 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T06:14:20.163 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T06:14:20.163 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T06:14:20.163 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T06:14:20.163 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T06:14:20.163 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T06:14:20.163 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T06:14:20.163 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:14:20.164 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:14:20.164 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:14:20.164 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.192403/ceph-mon.a.asok 2024-09-16T06:14:20.164 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T06:14:20.164 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.192403/ceph-mon.a.asok config get mon_host 2024-09-16T06:14:20.241 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:77: TEST_check_version_health_1: run_mon td/ver-health b --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-09-16T06:14:20.241 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-09-16T06:14:20.241 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T06:14:20.241 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-09-16T06:14:20.241 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T06:14:20.241 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/b 2024-09-16T06:14:20.242 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ver-health/b --run-dir=td/ver-health --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-09-16T06:14:20.918 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T06:14:20.918 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:14:20.918 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:14:20.919 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:14:20.919 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:14:20.919 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:14:20.919 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:14:20.919 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/b '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-09-16T06:14:21.221 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T06:14:21.222 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-09-16T06:14:21.222 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T06:14:21.223 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-16T06:14:21.223 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T06:14:21.223 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T06:14:21.223 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-16T06:14:21.223 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-16T06:14:21.223 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-16T06:14:21.224 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:14:21.224 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:14:21.224 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:14:21.224 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.192403/ceph-mon.b.asok 2024-09-16T06:14:21.224 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T06:14:21.224 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.192403/ceph-mon.b.asok config get fsid 2024-09-16T06:14:21.310 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-09-16T06:14:21.310 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T06:14:21.310 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-16T06:14:21.310 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T06:14:21.310 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T06:14:21.311 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-16T06:14:21.311 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-16T06:14:21.311 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-16T06:14:21.311 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:14:21.311 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:14:21.311 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:14:21.312 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.192403/ceph-mon.b.asok 2024-09-16T06:14:21.312 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T06:14:21.312 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.192403/ceph-mon.b.asok config get mon_host 2024-09-16T06:14:21.392 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:78: TEST_check_version_health_1: run_osd td/ver-health 0 2024-09-16T06:14:21.392 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-09-16T06:14:21.392 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T06:14:21.393 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T06:14:21.393 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T06:14:21.393 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/0 2024-09-16T06:14:21.393 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T06:14:21.393 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:14:21.393 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:14:21.393 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:14:21.393 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/0' 2024-09-16T06:14:21.393 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2024-09-16T06:14:21.393 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T06:14:21.393 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T06:14:21.393 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-16T06:14:21.394 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T06:14:21.394 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:14:21.394 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:14:21.394 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:14:21.394 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:14:21.394 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:14:21.394 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:14:21.394 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:14:21.394 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:14:21.394 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T06:14:21.394 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T06:14:21.394 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-16T06:14:21.394 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-16T06:14:21.394 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:14:21.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:14:21.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:14:21.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:14:21.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T06:14:21.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T06:14:21.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/0 2024-09-16T06:14:21.395 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T06:14:21.395 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 d5ca2365-8cf9-4b42-a3d3-198c6041ea25 2024-09-16T06:14:21.396 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d5ca2365-8cf9-4b42-a3d3-198c6041ea25 2024-09-16T06:14:21.396 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d5ca2365-8cf9-4b42-a3d3-198c6041ea25' 2024-09-16T06:14:21.396 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T06:14:21.410 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC9zOdmgtNyGBAAeGhn4enmGMwEINOK8oZCxQ== 2024-09-16T06:14:21.411 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC9zOdmgtNyGBAAeGhn4enmGMwEINOK8oZCxQ=="}' 2024-09-16T06:14:21.411 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d5ca2365-8cf9-4b42-a3d3-198c6041ea25 -i td/ver-health/0/new.json 2024-09-16T06:14:27.711 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:14:27.721 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/0/new.json 2024-09-16T06:14:27.722 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC9zOdmgtNyGBAAeGhn4enmGMwEINOK8oZCxQ== --osd-uuid d5ca2365-8cf9-4b42-a3d3-198c6041ea25 2024-09-16T06:14:27.751 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:14:27.750+0000 7f55ef341640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:14:27.753 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:14:27.753+0000 7f55ef341640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:14:27.754 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:14:27.754+0000 7f55ef341640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:14:27.754 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:14:27.754+0000 7f55ef341640 -1 bdev(0x5557c7c07c00 td/ver-health/0/block) open stat got: (1) Operation not permitted 2024-09-16T06:14:27.755 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:14:27.754+0000 7f55ef341640 -1 bluestore(td/ver-health/0) _read_fsid unparsable uuid 2024-09-16T06:14:33.880 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/0/keyring 2024-09-16T06:14:33.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T06:14:33.881 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T06:14:33.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T06:14:33.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T06:14:34.081 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T06:14:34.084 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T06:14:34.084 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T06:14:34.085 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T06:14:34.085 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T06:14:34.085 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T06:14:34.107 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:14:34.107+0000 7f84bac0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:14:34.113 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:14:34.113+0000 7f84bac0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:14:34.114 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:14:34.114+0000 7f84bac0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:14:34.316 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T06:14:34.316 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:14:34.316 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T06:14:34.316 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:14:34.316 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:14:34.317 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:14:34.317 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:14:34.317 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:14:34.317 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:14:34.317 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:14:34.688 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:14:35.689 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:14:35.716 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:14:35.716 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:14:35.717 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T06:14:35.717 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:14:35.717 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:14:35.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:14:35.950 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:14:35.950+0000 7f84bac0f640 -1 Falling back to public interface 2024-09-16T06:14:36.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:14:36.882 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:14:36.882 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:14:36.883 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T06:14:36.883 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:14:36.883 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:14:37.074 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:14:38.075 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:14:38.076 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:14:38.076 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T06:14:38.077 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T06:14:38.077 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:14:38.077 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:14:38.265 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:14:38.945 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:14:38.944+0000 7f84bac0f640 -1 osd.0 0 log_to_monitors true 2024-09-16T06:14:39.266 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:14:39.267 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:14:39.267 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T06:14:39.267 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T06:14:39.267 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:14:39.268 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:14:40.334 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:14:41.336 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:14:41.336 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:14:41.336 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T06:14:41.337 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T06:14:41.337 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:14:41.337 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:14:41.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:14:42.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:14:42.531 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:14:42.531 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T06:14:42.531 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T06:14:42.531 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:14:42.531 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:14:42.724 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:14:43.726 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:14:43.726 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:14:43.726 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T06:14:43.727 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T06:14:43.727 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:14:43.727 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:14:43.918 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:14:44.920 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:14:44.920 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:14:44.920 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T06:14:44.920 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T06:14:44.921 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:14:44.921 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:14:45.111 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:14:46.113 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:14:46.113 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:14:46.113 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T06:14:46.114 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T06:14:46.114 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:14:46.114 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:14:46.304 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:14:47.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:14:47.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:14:47.306 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T06:14:47.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T06:14:47.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:14:47.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:14:47.497 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:14:48.498 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:14:48.498 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:14:48.498 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T06:14:48.499 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T06:14:48.499 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:14:48.499 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:14:48.691 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:14:49.692 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:14:49.693 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:14:49.693 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T06:14:49.693 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T06:14:49.694 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:14:49.694 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:14:49.887 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:14:50.888 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:14:50.888 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:14:50.888 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T06:14:50.889 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T06:14:50.889 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:14:50.889 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:14:51.084 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:14:52.085 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:14:52.085 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:14:52.086 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T06:14:52.086 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T06:14:52.086 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:14:52.086 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:14:52.277 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:14:53.278 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:14:53.278 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:14:53.278 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T06:14:53.279 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T06:14:53.279 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:14:53.279 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:14:53.469 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:14:54.471 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:14:54.471 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:14:54.471 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T06:14:54.472 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T06:14:54.472 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:14:54.472 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:14:54.665 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:14:55.667 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:14:55.667 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:14:55.668 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T06:14:55.668 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T06:14:55.668 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:14:55.668 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:14:55.860 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:14:56.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:14:56.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:14:56.861 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T06:14:56.862 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T06:14:56.862 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:14:56.862 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:14:57.051 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:14:58.053 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:14:58.053 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:14:58.053 INFO:tasks.workunit.client.0.smithi178.stdout:19 2024-09-16T06:14:58.053 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-16T06:14:58.054 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:14:58.054 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:14:58.245 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:14:59.246 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:14:59.247 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:14:59.247 INFO:tasks.workunit.client.0.smithi178.stdout:20 2024-09-16T06:14:59.247 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-16T06:14:59.247 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:14:59.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:14:59.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:00.442 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:00.442 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:00.442 INFO:tasks.workunit.client.0.smithi178.stdout:21 2024-09-16T06:15:00.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-09-16T06:15:00.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:00.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:15:00.630 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:01.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:01.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:01.632 INFO:tasks.workunit.client.0.smithi178.stdout:22 2024-09-16T06:15:01.633 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-09-16T06:15:01.633 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:01.633 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:15:01.846 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:02.847 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:02.847 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:02.848 INFO:tasks.workunit.client.0.smithi178.stdout:23 2024-09-16T06:15:02.848 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-09-16T06:15:02.848 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:02.848 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:15:03.032 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3982921513,v1:127.0.0.1:6801/3982921513] [v2:127.0.0.1:6802/3982921513,v1:127.0.0.1:6803/3982921513] exists,up d5ca2365-8cf9-4b42-a3d3-198c6041ea25 2024-09-16T06:15:03.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:15:03.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:15:03.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:15:03.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:79: TEST_check_version_health_1: run_osd td/ver-health 1 2024-09-16T06:15:03.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-09-16T06:15:03.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T06:15:03.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T06:15:03.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T06:15:03.037 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/1 2024-09-16T06:15:03.037 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T06:15:03.037 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:15:03.037 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:15:03.037 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:15:03.037 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/1' 2024-09-16T06:15:03.037 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2024-09-16T06:15:03.037 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T06:15:03.037 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T06:15:03.037 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-16T06:15:03.037 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T06:15:03.037 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:15:03.037 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:15:03.038 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:15:03.038 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:15:03.038 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:15:03.038 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:15:03.038 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:15:03.038 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:15:03.038 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T06:15:03.038 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T06:15:03.038 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-16T06:15:03.038 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-16T06:15:03.038 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:15:03.038 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:15:03.038 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:15:03.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:15:03.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T06:15:03.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T06:15:03.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/1 2024-09-16T06:15:03.039 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T06:15:03.039 INFO:tasks.workunit.client.0.smithi178.stdout:add osd1 28a04514-868b-4991-a8a1-6676f5992d7b 2024-09-16T06:15:03.040 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=28a04514-868b-4991-a8a1-6676f5992d7b 2024-09-16T06:15:03.040 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 28a04514-868b-4991-a8a1-6676f5992d7b' 2024-09-16T06:15:03.040 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T06:15:03.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDnzOdm1gU7AxAAqKvkxSP7XbNEIYP2+Hlu7g== 2024-09-16T06:15:03.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDnzOdm1gU7AxAAqKvkxSP7XbNEIYP2+Hlu7g=="}' 2024-09-16T06:15:03.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 28a04514-868b-4991-a8a1-6676f5992d7b -i td/ver-health/1/new.json 2024-09-16T06:15:03.967 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:15:03.977 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/1/new.json 2024-09-16T06:15:03.978 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDnzOdm1gU7AxAAqKvkxSP7XbNEIYP2+Hlu7g== --osd-uuid 28a04514-868b-4991-a8a1-6676f5992d7b 2024-09-16T06:15:04.006 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:15:04.005+0000 7f481a62c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:15:04.052 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:15:04.052+0000 7f481a62c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:15:04.053 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:15:04.053+0000 7f481a62c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:15:04.054 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:15:04.053+0000 7f481a62c640 -1 bdev(0x56466b40dc00 td/ver-health/1/block) open stat got: (1) Operation not permitted 2024-09-16T06:15:04.054 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:15:04.053+0000 7f481a62c640 -1 bluestore(td/ver-health/1) _read_fsid unparsable uuid 2024-09-16T06:15:09.487 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/1/keyring 2024-09-16T06:15:09.487 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T06:15:09.488 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd1 key to auth repository 2024-09-16T06:15:09.489 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T06:15:09.489 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T06:15:09.691 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.1 2024-09-16T06:15:09.691 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T06:15:09.692 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T06:15:09.692 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T06:15:09.692 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T06:15:09.692 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T06:15:09.717 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:15:09.717+0000 7f6ceb233640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:15:09.720 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:15:09.720+0000 7f6ceb233640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:15:09.721 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:15:09.721+0000 7f6ceb233640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:15:09.915 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T06:15:09.915 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:15:09.915 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T06:15:09.915 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:15:09.915 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:15:09.915 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:09.916 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:15:09.916 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:15:09.916 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:09.916 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:15:10.143 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:11.144 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:11.144 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:11.144 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:15:11.145 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T06:15:11.145 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:11.145 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:15:11.314 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:15:11.314+0000 7f6ceb233640 -1 Falling back to public interface 2024-09-16T06:15:11.332 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:12.334 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:12.334 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:12.334 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T06:15:12.334 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:15:12.335 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:12.335 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:15:12.519 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:13.520 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:13.521 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:13.521 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T06:15:13.521 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T06:15:13.521 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:13.521 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:15:13.707 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:14.459 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:15:14.459+0000 7f6ceb233640 -1 osd.1 0 log_to_monitors true 2024-09-16T06:15:14.708 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:14.709 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:14.709 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T06:15:14.709 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T06:15:14.709 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:14.710 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:15:15.125 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:16.126 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:16.126 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:16.127 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T06:15:16.127 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T06:15:16.127 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:16.127 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:15:16.318 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:17.319 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:17.320 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:17.320 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T06:15:17.320 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T06:15:17.320 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:17.321 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:15:17.511 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:18.512 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:18.512 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:18.513 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T06:15:18.513 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T06:15:18.513 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:18.513 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:15:18.705 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:19.706 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:19.707 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:19.707 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T06:15:19.707 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T06:15:19.708 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:19.708 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:15:19.898 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:20.899 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:20.899 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:20.900 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T06:15:20.900 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T06:15:20.900 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:20.900 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:15:21.091 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:22.092 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:22.092 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:22.093 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T06:15:22.093 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T06:15:22.093 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:22.093 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:15:22.284 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:23.286 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:23.286 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:23.286 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T06:15:23.287 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T06:15:23.287 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:23.287 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:15:23.476 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:24.478 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:24.478 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T06:15:24.478 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:24.479 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T06:15:24.479 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:24.479 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:15:24.668 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:25.672 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:25.673 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:25.673 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T06:15:25.673 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T06:15:25.673 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:25.673 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:15:25.860 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:26.862 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:26.862 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:26.862 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T06:15:26.863 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T06:15:26.863 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:26.863 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:15:27.052 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:28.054 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:28.054 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:28.054 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T06:15:28.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T06:15:28.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:28.055 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:15:28.246 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:29.247 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:29.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:29.248 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T06:15:29.248 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T06:15:29.249 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:29.249 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:15:29.440 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:30.441 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:30.442 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:30.442 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T06:15:30.442 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T06:15:30.442 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:30.443 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:15:30.635 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:31.637 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:31.637 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:31.638 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T06:15:31.638 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T06:15:31.639 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:31.639 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:15:31.828 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:32.828 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:32.828 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:32.828 INFO:tasks.workunit.client.0.smithi178.stdout:19 2024-09-16T06:15:32.829 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-16T06:15:32.829 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:32.829 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:15:33.019 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:34.020 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:34.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:34.021 INFO:tasks.workunit.client.0.smithi178.stdout:20 2024-09-16T06:15:34.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-16T06:15:34.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:34.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:15:34.214 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:35.216 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:35.216 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:35.216 INFO:tasks.workunit.client.0.smithi178.stdout:21 2024-09-16T06:15:35.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-09-16T06:15:35.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:35.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:15:35.408 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:36.409 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:36.409 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:36.410 INFO:tasks.workunit.client.0.smithi178.stdout:22 2024-09-16T06:15:36.410 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-09-16T06:15:36.410 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:36.410 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:15:36.602 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:37.603 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:37.603 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:37.603 INFO:tasks.workunit.client.0.smithi178.stdout:23 2024-09-16T06:15:37.604 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-09-16T06:15:37.604 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:37.604 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:15:37.857 INFO:tasks.workunit.client.0.smithi178.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/2007531606,v1:127.0.0.1:6809/2007531606] [v2:127.0.0.1:6810/2007531606,v1:127.0.0.1:6811/2007531606] exists,up 28a04514-868b-4991-a8a1-6676f5992d7b 2024-09-16T06:15:37.860 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:15:37.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:15:37.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:15:37.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:80: TEST_check_version_health_1: run_osd td/ver-health 2 2024-09-16T06:15:37.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-09-16T06:15:37.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T06:15:37.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T06:15:37.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T06:15:37.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/2 2024-09-16T06:15:37.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T06:15:37.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:15:37.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:15:37.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:15:37.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/2' 2024-09-16T06:15:37.862 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2024-09-16T06:15:37.862 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T06:15:37.862 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T06:15:37.862 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-16T06:15:37.862 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T06:15:37.862 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:15:37.862 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:15:37.862 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:15:37.862 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:15:37.862 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:15:37.862 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:15:37.862 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:15:37.862 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:15:37.862 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T06:15:37.863 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T06:15:37.863 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-16T06:15:37.863 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-16T06:15:37.863 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:15:37.863 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:15:37.863 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:15:37.863 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:15:37.863 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T06:15:37.863 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T06:15:37.863 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/2 2024-09-16T06:15:37.863 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T06:15:37.864 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=576af293-cdad-4048-bd9c-359d50c17725 2024-09-16T06:15:37.864 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 576af293-cdad-4048-bd9c-359d50c17725' 2024-09-16T06:15:37.864 INFO:tasks.workunit.client.0.smithi178.stdout:add osd2 576af293-cdad-4048-bd9c-359d50c17725 2024-09-16T06:15:37.864 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T06:15:37.879 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAJzedmHrJfNBAAxdIPd5toGqvDpLbmbnncqA== 2024-09-16T06:15:37.879 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAJzedmHrJfNBAAxdIPd5toGqvDpLbmbnncqA=="}' 2024-09-16T06:15:37.879 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 576af293-cdad-4048-bd9c-359d50c17725 -i td/ver-health/2/new.json 2024-09-16T06:15:38.258 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:15:38.267 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/2/new.json 2024-09-16T06:15:38.268 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAJzedmHrJfNBAAxdIPd5toGqvDpLbmbnncqA== --osd-uuid 576af293-cdad-4048-bd9c-359d50c17725 2024-09-16T06:15:38.295 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:15:38.295+0000 7fe338d45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:15:38.297 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:15:38.297+0000 7fe338d45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:15:38.298 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:15:38.298+0000 7fe338d45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:15:38.299 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:15:38.298+0000 7fe338d45640 -1 bdev(0x5578f6c8bc00 td/ver-health/2/block) open stat got: (1) Operation not permitted 2024-09-16T06:15:38.299 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:15:38.298+0000 7fe338d45640 -1 bluestore(td/ver-health/2) _read_fsid unparsable uuid 2024-09-16T06:15:44.479 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/2/keyring 2024-09-16T06:15:44.479 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T06:15:44.479 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T06:15:44.480 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd2 key to auth repository 2024-09-16T06:15:44.480 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T06:15:44.679 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.2 2024-09-16T06:15:44.680 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T06:15:44.680 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T06:15:44.680 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T06:15:44.680 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T06:15:44.680 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T06:15:44.705 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:15:44.704+0000 7f78281c3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:15:44.711 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:15:44.710+0000 7f78281c3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:15:44.712 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:15:44.711+0000 7f78281c3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:15:44.926 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T06:15:44.926 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:15:44.927 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T06:15:44.927 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:15:44.927 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:15:44.927 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:44.927 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:15:44.927 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:15:44.928 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:44.928 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:15:45.156 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:46.157 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:46.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:46.158 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:15:46.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T06:15:46.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:46.158 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:15:46.328 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:15:46.327+0000 7f78281c3640 -1 Falling back to public interface 2024-09-16T06:15:46.351 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:47.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:47.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:47.353 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:15:47.354 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T06:15:47.354 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:47.354 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:15:47.540 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:48.541 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:48.541 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:48.542 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T06:15:48.542 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T06:15:48.542 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:48.542 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:15:48.733 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:49.167 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:15:49.167+0000 7f78281c3640 -1 osd.2 0 log_to_monitors true 2024-09-16T06:15:49.735 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:49.735 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:49.735 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T06:15:49.736 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T06:15:49.736 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:49.736 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:15:50.489 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:51.491 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:51.491 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:51.491 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T06:15:51.492 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T06:15:51.492 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:51.492 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:15:51.684 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:52.685 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:52.685 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:52.686 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T06:15:52.686 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T06:15:52.686 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:52.686 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:15:52.878 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:53.879 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:53.879 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:53.879 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T06:15:53.880 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T06:15:53.880 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:53.880 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:15:54.071 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:55.073 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:55.073 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:55.073 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T06:15:55.074 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T06:15:55.074 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:55.074 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:15:55.264 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:56.266 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:56.266 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:56.266 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T06:15:56.266 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T06:15:56.267 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:56.267 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:15:56.458 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:57.459 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:57.459 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:57.460 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T06:15:57.460 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T06:15:57.460 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:57.460 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:15:57.651 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:58.652 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:58.652 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:58.653 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T06:15:58.653 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T06:15:58.653 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:58.653 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:15:58.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:15:59.845 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:15:59.845 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:15:59.845 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T06:15:59.846 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T06:15:59.846 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:15:59.846 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:16:00.037 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:16:01.038 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:16:01.038 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:16:01.038 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T06:16:01.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T06:16:01.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:16:01.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:16:01.232 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:16:02.233 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:16:02.233 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:16:02.234 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T06:16:02.234 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T06:16:02.234 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:16:02.234 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:16:02.421 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:16:03.422 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:16:03.423 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:16:03.423 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T06:16:03.423 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T06:16:03.423 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:16:03.423 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:16:03.615 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:16:04.616 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:16:04.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:16:04.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T06:16:04.617 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T06:16:04.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:16:04.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:16:04.811 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:16:05.812 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:16:05.812 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:16:05.813 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T06:16:05.813 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T06:16:05.814 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:16:05.814 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:16:06.004 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:16:07.005 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:16:07.005 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:16:07.006 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T06:16:07.006 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T06:16:07.006 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:16:07.006 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:16:07.197 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:16:08.198 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:16:08.199 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:16:08.199 INFO:tasks.workunit.client.0.smithi178.stdout:19 2024-09-16T06:16:08.199 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-16T06:16:08.200 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:16:08.200 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:16:08.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:16:09.422 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:16:09.422 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:16:09.422 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-16T06:16:09.422 INFO:tasks.workunit.client.0.smithi178.stdout:20 2024-09-16T06:16:09.423 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:16:09.423 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:16:09.615 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:16:10.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:16:10.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:16:10.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-09-16T06:16:10.617 INFO:tasks.workunit.client.0.smithi178.stdout:21 2024-09-16T06:16:10.618 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:16:10.618 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:16:10.809 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:16:11.810 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:16:11.810 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:16:11.810 INFO:tasks.workunit.client.0.smithi178.stdout:22 2024-09-16T06:16:11.811 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-09-16T06:16:11.811 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:16:11.811 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:16:12.003 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:16:13.004 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:16:13.005 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:16:13.005 INFO:tasks.workunit.client.0.smithi178.stdout:23 2024-09-16T06:16:13.005 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-09-16T06:16:13.005 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:16:13.006 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:16:13.196 INFO:tasks.workunit.client.0.smithi178.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/1269009415,v1:127.0.0.1:6817/1269009415] [v2:127.0.0.1:6818/1269009415,v1:127.0.0.1:6819/1269009415] exists,up 576af293-cdad-4048-bd9c-359d50c17725 2024-09-16T06:16:13.196 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:16:13.197 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:16:13.197 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:16:13.197 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:82: TEST_check_version_health_1: sleep 5 2024-09-16T06:16:18.197 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:83: TEST_check_version_health_1: ceph health detail 2024-09-16T06:16:18.385 INFO:tasks.workunit.client.0.smithi178.stdout:HEALTH_OK 2024-09-16T06:16:18.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:85: TEST_check_version_health_1: ceph health detail 2024-09-16T06:16:18.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:85: TEST_check_version_health_1: grep DAEMON_OLD_VERSION 2024-09-16T06:16:18.707 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:87: TEST_check_version_health_1: kill_daemons td/ver-health KILL osd.1 2024-09-16T06:16:18.708 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:16:18.708 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:16:18.708 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:16:18.708 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:16:18.708 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:16:18.816 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:16:18.816 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:88: TEST_check_version_health_1: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-09-16T06:16:18.816 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:88: TEST_check_version_health_1: activate_osd td/ver-health 1 2024-09-16T06:16:18.817 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-09-16T06:16:18.817 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T06:16:18.817 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-16T06:16:18.817 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T06:16:18.817 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/1 2024-09-16T06:16:18.817 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T06:16:18.817 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:16:18.817 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:16:18.817 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:16:18.817 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/1' 2024-09-16T06:16:18.817 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2024-09-16T06:16:18.817 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T06:16:18.817 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T06:16:18.818 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-16T06:16:18.818 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T06:16:18.818 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:16:18.818 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:16:18.818 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:16:18.818 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:16:18.818 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:16:18.818 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:16:18.818 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:16:18.818 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:16:18.818 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-16T06:16:18.818 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-16T06:16:18.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:16:18.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:16:18.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:16:18.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:16:18.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T06:16:18.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T06:16:18.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/1 2024-09-16T06:16:18.819 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.1 2024-09-16T06:16:18.820 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-16T06:16:18.820 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T06:16:18.820 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/1/whoami 2024-09-16T06:16:18.820 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-16T06:16:18.821 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T06:16:18.821 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T06:16:18.821 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T06:16:18.843 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:16:18.842+0000 7f0a4e158640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:16:18.923 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:16:18.922+0000 7f0a4e158640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:16:18.924 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:16:18.924+0000 7f0a4e158640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:16:19.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-16T06:16:19.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:16:19.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T06:16:19.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:16:19.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:16:19.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:16:19.031 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:16:19.032 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:16:19.032 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:16:19.032 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:16:19.223 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:16:20.224 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:16:20.224 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:16:20.224 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:16:20.225 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T06:16:20.225 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:16:20.225 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:16:20.278 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:16:20.277+0000 7f0a4e158640 -1 Falling back to public interface 2024-09-16T06:16:20.418 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:16:21.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:16:21.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:16:21.420 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:16:21.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T06:16:21.420 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:16:21.421 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:16:21.604 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:16:22.603 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:16:22.604 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:16:22.604 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T06:16:22.605 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T06:16:22.605 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:16:22.605 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:16:22.794 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:16:23.624 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:16:23.623+0000 7f0a4e158640 -1 osd.1 15 log_to_monitors true 2024-09-16T06:16:23.796 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:16:23.796 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:16:23.796 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T06:16:23.797 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T06:16:23.797 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:16:23.797 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:16:24.080 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:16:25.081 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:16:25.081 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:16:25.082 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T06:16:25.082 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T06:16:25.082 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:16:25.082 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:16:25.268 INFO:tasks.workunit.client.0.smithi178.stdout:osd.1 up in weight 1 up_from 18 up_thru 0 down_at 16 last_clean_interval [10,15) [v2:127.0.0.1:6808/1199476704,v1:127.0.0.1:6809/1199476704] [v2:127.0.0.1:6810/1199476704,v1:127.0.0.1:6811/1199476704] exists,up 28a04514-868b-4991-a8a1-6676f5992d7b 2024-09-16T06:16:25.269 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:16:25.269 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:16:25.269 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:16:25.270 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:90: TEST_check_version_health_1: wait_for_health_string 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T06:16:25.270 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T06:16:25.270 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2024-09-16T06:16:25.270 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2024-09-16T06:16:25.270 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2024-09-16T06:16:25.270 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2024-09-16T06:16:25.270 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T06:16:25.270 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T06:16:25.270 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T06:16:25.563 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T06:16:26.564 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T06:16:26.564 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T06:16:26.564 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T06:16:26.564 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T06:16:26.757 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T06:16:27.759 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T06:16:27.759 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T06:16:27.759 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T06:16:27.759 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T06:16:27.953 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T06:16:28.954 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T06:16:28.955 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T06:16:28.955 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T06:16:28.955 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T06:16:29.150 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T06:16:30.152 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T06:16:30.152 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T06:16:30.152 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T06:16:30.152 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T06:16:30.350 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T06:16:31.351 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T06:16:31.351 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T06:16:31.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T06:16:31.352 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T06:16:31.549 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2024-09-16T06:16:31.549 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2024-09-16T06:16:31.549 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2024-09-16T06:16:31.550 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2024-09-16T06:16:31.550 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2024-09-16T06:16:31.550 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:92: TEST_check_version_health_1: ceph health detail 2024-09-16T06:16:31.765 INFO:tasks.workunit.client.0.smithi178.stdout:HEALTH_WARN There is a daemon running an older version of ceph; no active mgr 2024-09-16T06:16:31.766 INFO:tasks.workunit.client.0.smithi178.stdout:[WRN] DAEMON_OLD_VERSION: There is a daemon running an older version of ceph 2024-09-16T06:16:31.766 INFO:tasks.workunit.client.0.smithi178.stdout: osd.1 is running an older version of ceph: 01.00.00-gversion-test 2024-09-16T06:16:31.766 INFO:tasks.workunit.client.0.smithi178.stdout:[WRN] MGR_DOWN: no active mgr 2024-09-16T06:16:31.775 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:94: TEST_check_version_health_1: ceph health detail 2024-09-16T06:16:31.775 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:94: TEST_check_version_health_1: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T06:16:31.998 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:95: TEST_check_version_health_1: ceph health detail 2024-09-16T06:16:31.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:95: TEST_check_version_health_1: grep -q '^[[]WRN[]] DAEMON_OLD_VERSION: There is a daemon running an older version of ceph' 2024-09-16T06:16:32.225 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:96: TEST_check_version_health_1: ceph health detail 2024-09-16T06:16:32.225 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:96: TEST_check_version_health_1: grep -q 'osd.1 is running an older version of ceph: 01.00.00-gversion-test' 2024-09-16T06:16:32.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:98: TEST_check_version_health_1: kill_daemons td/ver-health KILL osd.2 2024-09-16T06:16:32.426 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:16:32.426 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:16:32.426 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:16:32.426 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:16:32.426 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:16:32.534 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:16:32.534 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:99: TEST_check_version_health_1: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-09-16T06:16:32.534 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:99: TEST_check_version_health_1: activate_osd td/ver-health 2 2024-09-16T06:16:32.534 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-09-16T06:16:32.534 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T06:16:32.534 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-16T06:16:32.534 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T06:16:32.534 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/2 2024-09-16T06:16:32.534 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T06:16:32.534 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:16:32.535 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:16:32.535 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:16:32.535 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/2' 2024-09-16T06:16:32.535 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2024-09-16T06:16:32.535 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T06:16:32.535 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T06:16:32.535 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-16T06:16:32.535 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T06:16:32.535 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:16:32.535 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:16:32.535 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:16:32.535 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:16:32.535 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:16:32.536 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:16:32.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:16:32.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:16:32.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-16T06:16:32.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-16T06:16:32.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:16:32.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:16:32.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:16:32.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:16:32.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T06:16:32.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T06:16:32.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/2 2024-09-16T06:16:32.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-16T06:16:32.537 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.2 2024-09-16T06:16:32.537 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T06:16:32.538 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/2/whoami 2024-09-16T06:16:32.538 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-16T06:16:32.538 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T06:16:32.538 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T06:16:32.538 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T06:16:32.559 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:16:32.559+0000 7fe63da22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:16:32.561 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:16:32.561+0000 7fe63da22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:16:32.562 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:16:32.562+0000 7fe63da22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:16:32.812 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-16T06:16:32.813 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:16:32.813 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T06:16:32.813 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:16:32.813 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:16:32.813 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:16:32.813 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:16:32.814 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:16:32.814 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:16:32.814 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:16:33.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:16:33.656 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:16:33.656+0000 7fe63da22640 -1 Falling back to public interface 2024-09-16T06:16:34.040 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:16:34.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:16:34.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T06:16:34.041 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:16:34.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:16:34.042 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:16:34.232 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:16:35.233 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:16:35.233 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:16:35.234 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T06:16:35.234 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:16:35.234 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:16:35.234 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:16:35.428 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:16:36.430 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:16:36.430 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:16:36.430 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T06:16:36.431 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T06:16:36.431 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:16:36.431 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:16:36.621 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:16:37.019 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:16:37.018+0000 7fe63da22640 -1 osd.2 18 log_to_monitors true 2024-09-16T06:16:37.622 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:16:37.623 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:16:37.623 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T06:16:37.623 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T06:16:37.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:16:37.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:16:37.887 INFO:tasks.workunit.client.0.smithi178.stdout:osd.2 up in weight 1 up_from 21 up_thru 0 down_at 19 last_clean_interval [14,18) [v2:127.0.0.1:6816/4156863425,v1:127.0.0.1:6817/4156863425] [v2:127.0.0.1:6818/4156863425,v1:127.0.0.1:6819/4156863425] exists,up 576af293-cdad-4048-bd9c-359d50c17725 2024-09-16T06:16:37.888 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:16:37.888 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:16:37.888 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:16:37.888 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:100: TEST_check_version_health_1: kill_daemons td/ver-health KILL osd.0 2024-09-16T06:16:37.888 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:16:37.888 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:16:37.888 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:16:37.888 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:16:37.888 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:16:37.996 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:16:37.996 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:101: TEST_check_version_health_1: ceph_debug_version_for_testing=02.00.00-gversion-test 2024-09-16T06:16:37.996 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:101: TEST_check_version_health_1: activate_osd td/ver-health 0 2024-09-16T06:16:37.997 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-09-16T06:16:37.997 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T06:16:37.997 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T06:16:37.997 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T06:16:37.997 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/0 2024-09-16T06:16:37.997 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T06:16:37.997 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:16:37.997 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:16:37.997 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:16:37.997 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/0' 2024-09-16T06:16:37.998 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2024-09-16T06:16:37.998 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T06:16:37.998 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T06:16:37.998 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-16T06:16:37.998 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T06:16:37.998 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:16:37.999 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:16:37.999 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:16:37.999 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:16:37.999 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:16:37.999 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:16:37.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:16:37.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:16:37.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-16T06:16:37.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-16T06:16:37.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:16:38.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:16:38.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:16:38.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:16:38.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T06:16:38.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T06:16:38.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/0 2024-09-16T06:16:38.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T06:16:38.000 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T06:16:38.002 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T06:16:38.002 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/0/whoami 2024-09-16T06:16:38.002 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T06:16:38.002 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T06:16:38.002 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T06:16:38.002 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T06:16:38.021 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:16:38.020+0000 7f3243927640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:16:38.023 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:16:38.022+0000 7f3243927640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:16:38.024 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:16:38.023+0000 7f3243927640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:16:38.329 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T06:16:38.330 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:16:38.330 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T06:16:38.330 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:16:38.330 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:16:38.330 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:16:38.330 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:16:38.331 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:16:38.331 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:16:38.331 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:16:38.845 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:16:39.625 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:16:39.624+0000 7f3243927640 -1 Falling back to public interface 2024-09-16T06:16:39.846 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:16:39.847 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:16:39.847 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:16:39.847 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T06:16:39.847 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:16:39.848 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:16:40.039 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:16:41.040 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:16:41.040 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:16:41.040 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:16:41.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T06:16:41.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:16:41.041 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:16:41.493 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:16:42.495 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:16:42.495 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:16:42.495 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T06:16:42.495 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T06:16:42.496 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:16:42.496 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:16:42.591 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:16:42.590+0000 7f3243927640 -1 osd.0 21 log_to_monitors true 2024-09-16T06:16:42.684 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:16:43.686 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:16:43.686 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:16:43.686 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T06:16:43.687 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T06:16:43.687 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:16:43.687 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:16:43.877 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6800/1453326541,v1:127.0.0.1:6801/1453326541] [v2:127.0.0.1:6802/1453326541,v1:127.0.0.1:6803/1453326541] exists,up d5ca2365-8cf9-4b42-a3d3-198c6041ea25 2024-09-16T06:16:43.878 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:16:43.879 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:16:43.879 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:16:43.879 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:103: TEST_check_version_health_1: wait_for_health_string 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-16T06:16:43.879 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-16T06:16:43.879 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2024-09-16T06:16:43.879 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2024-09-16T06:16:43.879 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2024-09-16T06:16:43.879 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2024-09-16T06:16:43.879 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T06:16:43.879 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T06:16:43.879 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-16T06:16:44.104 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T06:16:45.105 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T06:16:45.105 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T06:16:45.105 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T06:16:45.105 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-16T06:16:45.302 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T06:16:46.303 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T06:16:46.303 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T06:16:46.303 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T06:16:46.304 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-16T06:16:46.594 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2024-09-16T06:16:46.594 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2024-09-16T06:16:46.594 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2024-09-16T06:16:46.595 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2024-09-16T06:16:46.595 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2024-09-16T06:16:46.595 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:105: TEST_check_version_health_1: ceph health detail 2024-09-16T06:16:46.819 INFO:tasks.workunit.client.0.smithi178.stdout:HEALTH_ERR There are daemons running multiple old versions of ceph; no active mgr 2024-09-16T06:16:46.819 INFO:tasks.workunit.client.0.smithi178.stdout:[ERR] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph 2024-09-16T06:16:46.819 INFO:tasks.workunit.client.0.smithi178.stdout: osd.1 osd.2 are running an older version of ceph: 01.00.00-gversion-test 2024-09-16T06:16:46.819 INFO:tasks.workunit.client.0.smithi178.stdout: osd.0 is running an older version of ceph: 02.00.00-gversion-test 2024-09-16T06:16:46.819 INFO:tasks.workunit.client.0.smithi178.stdout:[WRN] MGR_DOWN: no active mgr 2024-09-16T06:16:46.828 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:106: TEST_check_version_health_1: ceph health detail 2024-09-16T06:16:46.829 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:106: TEST_check_version_health_1: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-16T06:16:47.080 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:107: TEST_check_version_health_1: ceph health detail 2024-09-16T06:16:47.080 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:107: TEST_check_version_health_1: grep -q '^[[]ERR[]] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph' 2024-09-16T06:16:47.308 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:108: TEST_check_version_health_1: ceph health detail 2024-09-16T06:16:47.308 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:108: TEST_check_version_health_1: grep -q 'osd.1 osd.2 are running an older version of ceph: 01.00.00-gversion-test' 2024-09-16T06:16:47.522 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:109: TEST_check_version_health_1: ceph health detail 2024-09-16T06:16:47.523 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:109: TEST_check_version_health_1: grep -q 'osd.0 is running an older version of ceph: 02.00.00-gversion-test' 2024-09-16T06:16:47.788 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:35: run: teardown td/ver-health 2024-09-16T06:16:47.788 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-16T06:16:47.788 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T06:16:47.788 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-16T06:16:47.788 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:16:47.788 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:16:47.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:16:47.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:16:47.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:16:47.895 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:16:47.896 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:16:47.896 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:16:47.896 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:16:47.897 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:16:47.898 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:16:47.898 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:16:47.899 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:16:47.899 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:16:47.899 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:16:47.899 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:16:47.899 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:16:47.900 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T06:16:47.901 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-16T06:16:47.916 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:16:47.916 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:16:47.916 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:16:47.916 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.192403 2024-09-16T06:16:47.918 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:16:47.918 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:16:47.918 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:32: run: for func in $funcs 2024-09-16T06:16:47.918 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:33: run: setup td/ver-health 2024-09-16T06:16:47.919 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2024-09-16T06:16:47.919 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2024-09-16T06:16:47.919 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-16T06:16:47.919 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T06:16:47.919 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-16T06:16:47.919 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:16:47.919 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:16:47.919 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:16:47.919 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:16:47.919 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:16:47.920 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:16:47.920 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:16:47.920 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:16:47.921 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:16:47.922 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:16:47.922 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:16:47.922 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:16:47.923 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:16:47.923 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:16:47.923 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:16:47.923 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:16:47.924 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:16:47.925 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T06:16:47.925 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-16T06:16:47.926 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:16:47.926 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:16:47.926 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:16:47.926 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.192403 2024-09-16T06:16:47.927 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:16:47.927 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:16:47.927 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2024-09-16T06:16:47.928 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T06:16:47.928 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:16:47.928 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:16:47.929 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.192403 2024-09-16T06:16:47.931 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T06:16:47.931 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T06:16:47.931 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T06:16:47.931 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2024-09-16T06:16:47.931 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:34: run: TEST_check_version_health_2 td/ver-health 2024-09-16T06:16:47.931 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:114: TEST_check_version_health_2: local dir=td/ver-health 2024-09-16T06:16:47.931 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:117: TEST_check_version_health_2: CEPH_ARGS='--fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T06:16:47.931 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:119: TEST_check_version_health_2: setup td/ver-health 2024-09-16T06:16:47.931 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2024-09-16T06:16:47.931 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2024-09-16T06:16:47.932 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-16T06:16:47.932 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T06:16:47.932 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-16T06:16:47.932 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:16:47.932 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:16:47.932 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:16:47.932 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:16:47.932 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:16:47.935 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:16:47.935 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:16:47.936 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:16:47.936 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:16:47.936 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:16:47.936 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:16:47.936 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:16:47.937 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:16:47.937 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:16:47.937 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:16:47.937 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:16:47.937 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:16:47.937 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T06:16:47.937 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-16T06:16:47.938 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:16:47.938 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:16:47.938 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:16:47.938 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.192403 2024-09-16T06:16:47.939 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:16:47.939 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:16:47.939 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2024-09-16T06:16:47.941 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T06:16:47.941 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:16:47.941 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:16:47.941 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.192403 2024-09-16T06:16:47.942 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T06:16:47.942 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T06:16:47.942 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T06:16:47.943 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2024-09-16T06:16:47.943 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:122: TEST_check_version_health_2: run_mon td/ver-health a --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-09-16T06:16:47.943 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-09-16T06:16:47.943 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T06:16:47.943 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T06:16:47.943 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T06:16:47.943 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/a 2024-09-16T06:16:47.943 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ver-health/a --run-dir=td/ver-health --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-09-16T06:16:48.327 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T06:16:48.327 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:16:48.327 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:16:48.327 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:16:48.327 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:16:48.327 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:16:48.327 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:16:48.327 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/a '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-09-16T06:16:48.527 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T06:16:48.529 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T06:16:48.529 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T06:16:48.529 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T06:16:48.529 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T06:16:48.529 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T06:16:48.529 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T06:16:48.529 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T06:16:48.530 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T06:16:48.530 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:16:48.530 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:16:48.530 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:16:48.530 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.192403/ceph-mon.a.asok 2024-09-16T06:16:48.530 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T06:16:48.530 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.192403/ceph-mon.a.asok config get fsid 2024-09-16T06:16:48.609 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T06:16:48.609 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T06:16:48.609 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T06:16:48.609 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T06:16:48.609 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T06:16:48.609 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T06:16:48.609 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T06:16:48.609 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T06:16:48.610 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:16:48.610 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:16:48.610 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:16:48.610 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.192403/ceph-mon.a.asok 2024-09-16T06:16:48.610 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T06:16:48.610 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.192403/ceph-mon.a.asok config get mon_host 2024-09-16T06:16:48.688 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:123: TEST_check_version_health_2: run_mon td/ver-health b --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-09-16T06:16:48.688 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-09-16T06:16:48.688 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T06:16:48.688 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-09-16T06:16:48.688 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T06:16:48.688 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/b 2024-09-16T06:16:48.689 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ver-health/b --run-dir=td/ver-health --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-09-16T06:16:49.382 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T06:16:49.382 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:16:49.382 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:16:49.383 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:16:49.383 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:16:49.383 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:16:49.383 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:16:49.383 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/b '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-09-16T06:16:49.644 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T06:16:49.645 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-09-16T06:16:49.645 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T06:16:49.645 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-16T06:16:49.645 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T06:16:49.647 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T06:16:49.647 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-16T06:16:49.647 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-16T06:16:49.647 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-16T06:16:49.647 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:16:49.647 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:16:49.647 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:16:49.647 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.192403/ceph-mon.b.asok 2024-09-16T06:16:49.647 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T06:16:49.647 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.192403/ceph-mon.b.asok config get fsid 2024-09-16T06:16:49.735 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-09-16T06:16:49.735 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T06:16:49.735 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-16T06:16:49.735 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T06:16:49.735 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T06:16:49.735 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-16T06:16:49.736 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-16T06:16:49.736 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-16T06:16:49.736 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:16:49.736 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:16:49.736 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:16:49.736 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.192403/ceph-mon.b.asok 2024-09-16T06:16:49.736 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T06:16:49.736 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.192403/ceph-mon.b.asok config get mon_host 2024-09-16T06:16:49.817 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:124: TEST_check_version_health_2: run_osd td/ver-health 0 2024-09-16T06:16:49.818 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-09-16T06:16:49.818 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T06:16:49.818 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T06:16:49.818 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T06:16:49.818 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/0 2024-09-16T06:16:49.818 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T06:16:49.818 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:16:49.818 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:16:49.818 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:16:49.818 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/0' 2024-09-16T06:16:49.818 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2024-09-16T06:16:49.818 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T06:16:49.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T06:16:49.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-16T06:16:49.819 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T06:16:49.819 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:16:49.819 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:16:49.819 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:16:49.819 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:16:49.819 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:16:49.819 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:16:49.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:16:49.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:16:49.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T06:16:49.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T06:16:49.820 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-16T06:16:49.820 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-16T06:16:49.820 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:16:49.820 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:16:49.820 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:16:49.820 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:16:49.820 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T06:16:49.820 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T06:16:49.820 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/0 2024-09-16T06:16:49.820 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T06:16:49.820 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fc103cfa-4066-4971-96da-7da607e2104f 2024-09-16T06:16:49.820 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 fc103cfa-4066-4971-96da-7da607e2104f' 2024-09-16T06:16:49.821 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 fc103cfa-4066-4971-96da-7da607e2104f 2024-09-16T06:16:49.821 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T06:16:49.835 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBRzedm+6nCMRAARngk8eEgYeksutB3FQhKHA== 2024-09-16T06:16:49.836 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBRzedm+6nCMRAARngk8eEgYeksutB3FQhKHA=="}' 2024-09-16T06:16:49.836 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fc103cfa-4066-4971-96da-7da607e2104f -i td/ver-health/0/new.json 2024-09-16T06:16:56.134 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:16:56.145 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/0/new.json 2024-09-16T06:16:56.146 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBRzedm+6nCMRAARngk8eEgYeksutB3FQhKHA== --osd-uuid fc103cfa-4066-4971-96da-7da607e2104f 2024-09-16T06:16:56.175 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:16:56.174+0000 7fb5debb0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:16:56.177 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:16:56.176+0000 7fb5debb0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:16:56.178 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:16:56.177+0000 7fb5debb0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:16:56.178 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:16:56.178+0000 7fb5debb0640 -1 bdev(0x56349d6e3c00 td/ver-health/0/block) open stat got: (1) Operation not permitted 2024-09-16T06:16:56.179 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:16:56.178+0000 7fb5debb0640 -1 bluestore(td/ver-health/0) _read_fsid unparsable uuid 2024-09-16T06:17:01.844 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/0/keyring 2024-09-16T06:17:01.844 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T06:17:01.845 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T06:17:01.845 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T06:17:01.846 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T06:17:02.047 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T06:17:02.048 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T06:17:02.048 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T06:17:02.048 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T06:17:02.048 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T06:17:02.048 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T06:17:02.074 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:17:02.073+0000 7f109e01b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:17:02.080 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:17:02.080+0000 7f109e01b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:17:02.081 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:17:02.081+0000 7f109e01b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:17:02.274 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T06:17:02.274 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:17:02.274 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T06:17:02.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:17:02.275 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:17:02.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:17:02.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:02.276 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:17:02.276 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:02.276 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:17:02.648 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:03.444 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:17:03.443+0000 7f109e01b640 -1 Falling back to public interface 2024-09-16T06:17:03.649 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:03.649 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:03.649 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:17:03.650 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T06:17:03.650 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:03.650 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:17:03.842 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:04.843 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:04.844 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:04.844 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T06:17:04.844 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:17:04.844 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:04.845 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:17:05.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:06.035 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:06.035 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:06.036 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T06:17:06.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T06:17:06.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:06.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:17:06.103 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:17:06.102+0000 7f109e01b640 -1 osd.0 0 log_to_monitors true 2024-09-16T06:17:06.235 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:07.237 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:07.237 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:07.237 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T06:17:07.238 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T06:17:07.238 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:07.238 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:17:07.575 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:08.576 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:08.577 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:08.577 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T06:17:08.577 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T06:17:08.578 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:08.578 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:17:08.768 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:09.769 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:09.769 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:09.770 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T06:17:09.770 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T06:17:09.770 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:09.770 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:17:09.962 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:10.963 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:10.964 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:10.964 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T06:17:10.964 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T06:17:10.964 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:10.964 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:17:11.159 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:12.160 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:12.161 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:12.161 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T06:17:12.161 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T06:17:12.161 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:12.161 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:17:12.353 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:13.354 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:13.354 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:13.355 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T06:17:13.355 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T06:17:13.355 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:13.355 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:17:13.547 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:14.548 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:14.548 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:14.549 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T06:17:14.549 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T06:17:14.549 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:14.549 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:17:14.742 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:15.743 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:15.743 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:15.743 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T06:17:15.744 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T06:17:15.744 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:15.744 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:17:15.935 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:16.936 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:16.936 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:16.937 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T06:17:16.937 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T06:17:16.937 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:16.937 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:17:17.130 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:18.131 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:18.132 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T06:17:18.132 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:18.133 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T06:17:18.133 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:18.133 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:17:18.321 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:19.323 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:19.323 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:19.323 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T06:17:19.324 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T06:17:19.324 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:19.324 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:17:19.516 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:20.517 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:20.517 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:20.518 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T06:17:20.518 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T06:17:20.518 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:20.518 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:17:20.709 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:21.711 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:21.711 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:21.711 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T06:17:21.712 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T06:17:21.712 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:21.712 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:17:21.905 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:22.906 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:22.907 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T06:17:22.907 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:22.907 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T06:17:22.907 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:22.907 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:17:23.101 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:24.102 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:24.102 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:24.103 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T06:17:24.103 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T06:17:24.103 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:24.103 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:17:24.290 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:25.292 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:25.292 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:25.292 INFO:tasks.workunit.client.0.smithi178.stdout:19 2024-09-16T06:17:25.293 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-16T06:17:25.293 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:25.293 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:17:25.533 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:26.534 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:26.535 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:26.535 INFO:tasks.workunit.client.0.smithi178.stdout:20 2024-09-16T06:17:26.535 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-16T06:17:26.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:26.536 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:17:26.718 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:27.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:27.719 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:27.720 INFO:tasks.workunit.client.0.smithi178.stdout:21 2024-09-16T06:17:27.720 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-09-16T06:17:27.720 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:27.720 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:17:27.912 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:28.913 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:28.913 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:28.914 INFO:tasks.workunit.client.0.smithi178.stdout:22 2024-09-16T06:17:28.914 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-09-16T06:17:28.914 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:28.914 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:17:29.107 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:30.108 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:30.108 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:30.109 INFO:tasks.workunit.client.0.smithi178.stdout:23 2024-09-16T06:17:30.109 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-09-16T06:17:30.109 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:30.109 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:17:30.301 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/516036730,v1:127.0.0.1:6801/516036730] [v2:127.0.0.1:6802/516036730,v1:127.0.0.1:6803/516036730] exists,up fc103cfa-4066-4971-96da-7da607e2104f 2024-09-16T06:17:30.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:17:30.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:17:30.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:17:30.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:125: TEST_check_version_health_2: run_osd td/ver-health 1 2024-09-16T06:17:30.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-09-16T06:17:30.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T06:17:30.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T06:17:30.306 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T06:17:30.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/1 2024-09-16T06:17:30.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T06:17:30.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:17:30.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:17:30.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:17:30.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/1' 2024-09-16T06:17:30.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2024-09-16T06:17:30.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T06:17:30.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T06:17:30.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-16T06:17:30.307 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T06:17:30.307 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:17:30.308 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:17:30.308 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:17:30.308 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:17:30.308 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:17:30.308 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:17:30.308 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:17:30.308 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:17:30.308 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T06:17:30.308 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T06:17:30.308 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-16T06:17:30.309 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-16T06:17:30.309 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:17:30.309 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:17:30.309 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:17:30.309 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:17:30.309 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T06:17:30.309 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T06:17:30.309 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/1 2024-09-16T06:17:30.309 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T06:17:30.309 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5c913414-0bcc-4572-86f0-6987d947a4fa 2024-09-16T06:17:30.309 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 5c913414-0bcc-4572-86f0-6987d947a4fa' 2024-09-16T06:17:30.310 INFO:tasks.workunit.client.0.smithi178.stdout:add osd1 5c913414-0bcc-4572-86f0-6987d947a4fa 2024-09-16T06:17:30.310 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T06:17:30.323 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB6zedmdV09ExAAgmfBSesKi74RgSxAn2E1rg== 2024-09-16T06:17:30.323 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB6zedmdV09ExAAgmfBSesKi74RgSxAn2E1rg=="}' 2024-09-16T06:17:30.323 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5c913414-0bcc-4572-86f0-6987d947a4fa -i td/ver-health/1/new.json 2024-09-16T06:17:30.665 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:17:30.674 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/1/new.json 2024-09-16T06:17:30.675 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB6zedmdV09ExAAgmfBSesKi74RgSxAn2E1rg== --osd-uuid 5c913414-0bcc-4572-86f0-6987d947a4fa 2024-09-16T06:17:30.704 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:17:30.702+0000 7fa8ccbdb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:17:30.706 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:17:30.704+0000 7fa8ccbdb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:17:30.707 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:17:30.705+0000 7fa8ccbdb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:17:30.707 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:17:30.706+0000 7fa8ccbdb640 -1 bdev(0x5591ec4e7c00 td/ver-health/1/block) open stat got: (1) Operation not permitted 2024-09-16T06:17:30.707 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:17:30.706+0000 7fa8ccbdb640 -1 bluestore(td/ver-health/1) _read_fsid unparsable uuid 2024-09-16T06:17:36.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/1/keyring 2024-09-16T06:17:36.625 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T06:17:36.625 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd1 key to auth repository 2024-09-16T06:17:36.626 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T06:17:36.626 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T06:17:36.827 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.1 2024-09-16T06:17:36.827 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T06:17:36.827 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T06:17:36.828 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T06:17:36.828 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T06:17:36.828 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T06:17:36.853 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:17:36.852+0000 7fbb61a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:17:36.895 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:17:36.895+0000 7fbb61a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:17:36.897 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:17:36.896+0000 7fbb61a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:17:37.098 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T06:17:37.098 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:17:37.098 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T06:17:37.098 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:17:37.098 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:17:37.098 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:37.099 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:17:37.099 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:17:37.099 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:37.099 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:17:37.313 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:38.314 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:38.314 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:38.314 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:17:38.315 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T06:17:38.315 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:38.315 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:17:38.487 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:17:38.487+0000 7fbb61a0d640 -1 Falling back to public interface 2024-09-16T06:17:38.506 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:39.507 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:39.507 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:39.508 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:17:39.508 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T06:17:39.508 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:39.508 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:17:39.698 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:40.699 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:40.699 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:40.699 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T06:17:40.700 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T06:17:40.700 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:40.700 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:17:40.889 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:41.535 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:17:41.535+0000 7fbb61a0d640 -1 osd.1 0 log_to_monitors true 2024-09-16T06:17:41.890 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:41.891 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:41.891 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T06:17:41.891 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T06:17:41.891 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:41.891 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:17:42.230 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:43.231 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:43.232 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:43.232 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T06:17:43.232 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T06:17:43.232 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:43.233 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:17:43.423 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:44.424 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:44.425 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T06:17:44.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:44.425 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T06:17:44.426 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:44.426 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:17:44.617 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:45.619 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:45.619 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:45.619 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T06:17:45.620 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T06:17:45.620 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:45.620 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:17:45.814 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:46.815 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:46.816 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:46.816 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T06:17:46.816 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T06:17:46.817 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:46.817 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:17:47.008 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:48.009 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:48.009 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:48.010 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T06:17:48.010 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T06:17:48.010 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:48.010 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:17:48.201 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:49.202 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:49.203 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:49.203 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T06:17:49.203 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T06:17:49.203 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:49.204 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:17:49.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:50.396 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:50.396 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:50.397 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T06:17:50.397 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T06:17:50.397 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:50.397 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:17:50.589 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:51.590 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:51.591 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:51.591 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T06:17:51.591 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T06:17:51.591 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:51.591 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:17:51.781 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:52.783 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:52.783 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:52.783 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T06:17:52.783 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T06:17:52.783 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:52.784 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:17:52.974 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:53.976 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:53.976 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:53.976 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T06:17:53.977 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T06:17:53.977 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:53.977 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:17:54.169 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:55.170 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:55.170 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:55.170 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T06:17:55.170 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T06:17:55.171 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:55.171 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:17:55.388 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:56.389 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:56.390 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:56.390 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T06:17:56.390 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T06:17:56.390 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:56.391 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:17:56.582 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:57.584 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:57.584 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:57.584 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T06:17:57.585 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T06:17:57.585 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:57.585 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:17:57.777 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:58.778 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:58.778 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:58.778 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T06:17:58.779 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T06:17:58.779 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:58.779 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:17:58.970 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:17:59.972 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:17:59.972 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:17:59.972 INFO:tasks.workunit.client.0.smithi178.stdout:19 2024-09-16T06:17:59.973 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-16T06:17:59.973 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:17:59.973 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:18:00.167 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:18:01.168 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:18:01.168 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:18:01.168 INFO:tasks.workunit.client.0.smithi178.stdout:20 2024-09-16T06:18:01.169 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-16T06:18:01.169 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:18:01.169 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:18:01.359 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:18:02.361 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:18:02.361 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:18:02.361 INFO:tasks.workunit.client.0.smithi178.stdout:21 2024-09-16T06:18:02.361 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-09-16T06:18:02.362 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:18:02.362 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:18:02.551 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:18:03.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:18:03.553 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:18:03.553 INFO:tasks.workunit.client.0.smithi178.stdout:22 2024-09-16T06:18:03.554 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-09-16T06:18:03.554 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:18:03.554 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:18:03.745 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:18:04.746 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:18:04.746 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:18:04.747 INFO:tasks.workunit.client.0.smithi178.stdout:23 2024-09-16T06:18:04.747 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-09-16T06:18:04.747 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:18:04.747 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:18:05.030 INFO:tasks.workunit.client.0.smithi178.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/1567678360,v1:127.0.0.1:6809/1567678360] [v2:127.0.0.1:6810/1567678360,v1:127.0.0.1:6811/1567678360] exists,up 5c913414-0bcc-4572-86f0-6987d947a4fa 2024-09-16T06:18:05.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:18:05.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:18:05.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:18:05.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:126: TEST_check_version_health_2: run_osd td/ver-health 2 2024-09-16T06:18:05.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-09-16T06:18:05.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T06:18:05.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T06:18:05.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T06:18:05.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/2 2024-09-16T06:18:05.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T06:18:05.034 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:18:05.035 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:18:05.035 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:18:05.035 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/2' 2024-09-16T06:18:05.035 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2024-09-16T06:18:05.035 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T06:18:05.035 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T06:18:05.035 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-16T06:18:05.035 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T06:18:05.035 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:18:05.035 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:18:05.035 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:18:05.035 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:18:05.035 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:18:05.035 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:18:05.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:18:05.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:18:05.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T06:18:05.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T06:18:05.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-16T06:18:05.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-16T06:18:05.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:18:05.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:18:05.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:18:05.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:18:05.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T06:18:05.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T06:18:05.036 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/2 2024-09-16T06:18:05.036 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T06:18:05.037 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=27c1c00d-2693-48c1-b68b-6821d0e79563 2024-09-16T06:18:05.037 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 27c1c00d-2693-48c1-b68b-6821d0e79563' 2024-09-16T06:18:05.037 INFO:tasks.workunit.client.0.smithi178.stdout:add osd2 27c1c00d-2693-48c1-b68b-6821d0e79563 2024-09-16T06:18:05.037 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T06:18:05.051 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCdzedm+7sIAxAAOp30rMn4LEcC1fisSn2kQw== 2024-09-16T06:18:05.051 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCdzedm+7sIAxAAOp30rMn4LEcC1fisSn2kQw=="}' 2024-09-16T06:18:05.051 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 27c1c00d-2693-48c1-b68b-6821d0e79563 -i td/ver-health/2/new.json 2024-09-16T06:18:05.935 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:18:05.945 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/2/new.json 2024-09-16T06:18:05.946 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCdzedm+7sIAxAAOp30rMn4LEcC1fisSn2kQw== --osd-uuid 27c1c00d-2693-48c1-b68b-6821d0e79563 2024-09-16T06:18:05.974 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:18:05.973+0000 7f02dfbd9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:18:05.976 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:18:05.975+0000 7f02dfbd9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:18:05.978 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:18:05.977+0000 7f02dfbd9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:18:05.978 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:18:05.978+0000 7f02dfbd9640 -1 bdev(0x55e949b0bc00 td/ver-health/2/block) open stat got: (1) Operation not permitted 2024-09-16T06:18:05.978 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:18:05.978+0000 7f02dfbd9640 -1 bluestore(td/ver-health/2) _read_fsid unparsable uuid 2024-09-16T06:18:12.184 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/2/keyring 2024-09-16T06:18:12.184 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T06:18:12.185 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd2 key to auth repository 2024-09-16T06:18:12.186 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T06:18:12.186 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T06:18:12.386 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T06:18:12.387 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.2 2024-09-16T06:18:12.387 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T06:18:12.387 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T06:18:12.387 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T06:18:12.387 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T06:18:12.413 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:18:12.412+0000 7f183ca13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:18:12.417 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:18:12.416+0000 7f183ca13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:18:12.418 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:18:12.417+0000 7f183ca13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:18:12.611 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T06:18:12.611 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:18:12.611 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T06:18:12.611 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:18:12.612 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:18:12.612 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:18:12.612 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:18:12.612 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:18:12.612 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:18:12.612 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:18:12.830 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:18:13.498 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:18:13.498+0000 7f183ca13640 -1 Falling back to public interface 2024-09-16T06:18:13.831 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:18:13.831 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:18:13.832 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:18:13.832 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T06:18:13.832 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:18:13.832 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:18:14.015 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:18:15.017 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:18:15.017 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:18:15.017 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:18:15.018 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T06:18:15.018 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:18:15.018 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:18:15.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:18:16.139 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:18:16.139+0000 7f183ca13640 -1 osd.2 0 log_to_monitors true 2024-09-16T06:18:16.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:18:16.209 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T06:18:16.209 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:18:16.209 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T06:18:16.209 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:18:16.210 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:18:16.622 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:18:17.623 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:18:17.623 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:18:17.624 INFO:tasks.workunit.client.0.smithi178.stdout:4 2024-09-16T06:18:17.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T06:18:17.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:18:17.624 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:18:17.817 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:18:18.818 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:18:18.818 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:18:18.819 INFO:tasks.workunit.client.0.smithi178.stdout:5 2024-09-16T06:18:18.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T06:18:18.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:18:18.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:18:19.012 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:18:20.013 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:18:20.013 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:18:20.014 INFO:tasks.workunit.client.0.smithi178.stdout:6 2024-09-16T06:18:20.014 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T06:18:20.014 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:18:20.014 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:18:20.206 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:18:21.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:18:21.207 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:18:21.208 INFO:tasks.workunit.client.0.smithi178.stdout:7 2024-09-16T06:18:21.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T06:18:21.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:18:21.208 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:18:21.399 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:18:22.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:18:22.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:18:22.401 INFO:tasks.workunit.client.0.smithi178.stdout:8 2024-09-16T06:18:22.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T06:18:22.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:18:22.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:18:22.593 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:18:23.594 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:18:23.595 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:18:23.595 INFO:tasks.workunit.client.0.smithi178.stdout:9 2024-09-16T06:18:23.595 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T06:18:23.595 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:18:23.596 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:18:23.789 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:18:24.791 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:18:24.791 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:18:24.791 INFO:tasks.workunit.client.0.smithi178.stdout:10 2024-09-16T06:18:24.792 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T06:18:24.792 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:18:24.792 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:18:24.982 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:18:25.984 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:18:25.984 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:18:25.984 INFO:tasks.workunit.client.0.smithi178.stdout:11 2024-09-16T06:18:25.985 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T06:18:25.985 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:18:25.985 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:18:26.178 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:18:27.179 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:18:27.179 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:18:27.179 INFO:tasks.workunit.client.0.smithi178.stdout:12 2024-09-16T06:18:27.180 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T06:18:27.180 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:18:27.180 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:18:27.374 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:18:28.375 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:18:28.376 INFO:tasks.workunit.client.0.smithi178.stdout:13 2024-09-16T06:18:28.376 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:18:28.376 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T06:18:28.376 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:18:28.376 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:18:28.566 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:18:29.568 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:18:29.568 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:18:29.568 INFO:tasks.workunit.client.0.smithi178.stdout:14 2024-09-16T06:18:29.569 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T06:18:29.569 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:18:29.569 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:18:29.759 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:18:30.760 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:18:30.760 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:18:30.760 INFO:tasks.workunit.client.0.smithi178.stdout:15 2024-09-16T06:18:30.761 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T06:18:30.761 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:18:30.761 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:18:30.953 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:18:31.954 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:18:31.954 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:18:31.955 INFO:tasks.workunit.client.0.smithi178.stdout:16 2024-09-16T06:18:31.955 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T06:18:31.955 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:18:31.955 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:18:32.148 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:18:33.149 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:18:33.149 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:18:33.150 INFO:tasks.workunit.client.0.smithi178.stdout:17 2024-09-16T06:18:33.150 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T06:18:33.150 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:18:33.150 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:18:33.340 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:18:34.342 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:18:34.342 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:18:34.342 INFO:tasks.workunit.client.0.smithi178.stdout:18 2024-09-16T06:18:34.343 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T06:18:34.343 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:18:34.343 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:18:34.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:18:35.533 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:18:35.533 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:18:35.534 INFO:tasks.workunit.client.0.smithi178.stdout:19 2024-09-16T06:18:35.534 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-16T06:18:35.534 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:18:35.534 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:18:35.727 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:18:36.729 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:18:36.729 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:18:36.729 INFO:tasks.workunit.client.0.smithi178.stdout:20 2024-09-16T06:18:36.730 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-16T06:18:36.730 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:18:36.730 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:18:36.922 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:18:37.923 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:18:37.924 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:18:37.924 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-09-16T06:18:37.924 INFO:tasks.workunit.client.0.smithi178.stdout:21 2024-09-16T06:18:37.925 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:18:37.925 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:18:38.194 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:18:39.195 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:18:39.196 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:18:39.196 INFO:tasks.workunit.client.0.smithi178.stdout:22 2024-09-16T06:18:39.196 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-09-16T06:18:39.197 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:18:39.197 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:18:39.879 INFO:tasks.workunit.client.0.smithi178.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/1933396219,v1:127.0.0.1:6817/1933396219] [v2:127.0.0.1:6818/1933396219,v1:127.0.0.1:6819/1933396219] exists,up 27c1c00d-2693-48c1-b68b-6821d0e79563 2024-09-16T06:18:39.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:18:39.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:18:39.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:18:39.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:127: TEST_check_version_health_2: run_mgr td/ver-health x 2024-09-16T06:18:39.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ver-health 2024-09-16T06:18:39.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T06:18:39.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T06:18:39.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T06:18:39.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ver-health/x 2024-09-16T06:18:39.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T06:18:40.209 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T06:18:40.210 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:18:40.210 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:18:40.210 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:18:40.210 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:18:40.210 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:18:40.210 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:18:40.210 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T06:18:40.211 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ver-health/x '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T06:18:40.239 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:128: TEST_check_version_health_2: run_mgr td/ver-health y 2024-09-16T06:18:40.240 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ver-health 2024-09-16T06:18:40.240 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T06:18:40.240 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=y 2024-09-16T06:18:40.240 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T06:18:40.240 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ver-health/y 2024-09-16T06:18:40.240 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T06:18:40.496 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T06:18:40.496 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:18:40.496 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:18:40.496 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:18:40.496 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:18:40.497 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:18:40.497 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:18:40.497 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T06:18:40.497 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id y --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ver-health/y '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T06:18:40.527 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:129: TEST_check_version_health_2: run_mds td/ver-health m 2024-09-16T06:18:40.527 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ver-health 2024-09-16T06:18:40.527 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-09-16T06:18:40.527 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=m 2024-09-16T06:18:40.527 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-09-16T06:18:40.527 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ver-health/m 2024-09-16T06:18:40.527 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-09-16T06:18:40.527 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:18:40.528 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:18:40.528 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:18:40.528 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:18:40.528 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:18:40.528 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:18:40.528 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id m --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ver-health/m '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' 2024-09-16T06:18:40.553 INFO:tasks.workunit.client.0.smithi178.stdout:starting mds.m at 2024-09-16T06:18:40.557 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:130: TEST_check_version_health_2: run_mds td/ver-health n 2024-09-16T06:18:40.558 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ver-health 2024-09-16T06:18:40.558 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-09-16T06:18:40.558 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=n 2024-09-16T06:18:40.558 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-09-16T06:18:40.558 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ver-health/n 2024-09-16T06:18:40.558 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-09-16T06:18:40.558 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:18:40.558 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:18:40.558 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:18:40.558 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:18:40.558 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:18:40.558 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:18:40.559 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id n --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ver-health/n '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' 2024-09-16T06:18:40.643 INFO:tasks.workunit.client.0.smithi178.stdout:starting mds.n at 2024-09-16T06:18:40.647 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:132: TEST_check_version_health_2: sleep 5 2024-09-16T06:18:45.648 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:133: TEST_check_version_health_2: ceph health detail 2024-09-16T06:18:46.232 INFO:tasks.workunit.client.0.smithi178.stdout:HEALTH_OK 2024-09-16T06:18:46.243 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:135: TEST_check_version_health_2: ceph health detail 2024-09-16T06:18:46.243 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:135: TEST_check_version_health_2: grep DAEMON_OLD_VERSION 2024-09-16T06:18:46.829 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:137: TEST_check_version_health_2: kill_daemons td/ver-health KILL mon.b 2024-09-16T06:18:46.829 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:18:46.829 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:18:46.829 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:18:46.830 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:18:46.830 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:18:46.935 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:18:46.935 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:138: TEST_check_version_health_2: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-09-16T06:18:46.935 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:138: TEST_check_version_health_2: run_mon td/ver-health b --mon_warn_older_version_delay=0 2024-09-16T06:18:46.935 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-09-16T06:18:46.935 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T06:18:46.935 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-09-16T06:18:46.935 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T06:18:46.935 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/b 2024-09-16T06:18:46.935 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ver-health/b --run-dir=td/ver-health --mon_warn_older_version_delay=0 2024-09-16T06:18:46.956 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:18:46.956+0000 7f136ca3fc80 -1 'td/ver-health/b' already exists and is not empty: monitor may already exist 2024-09-16T06:18:46.958 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T06:18:46.958 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:18:46.959 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:18:46.959 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:18:46.959 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:18:46.959 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:18:46.959 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:18:46.959 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/b '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_warn_older_version_delay=0 2024-09-16T06:18:47.149 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T06:18:47.151 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-09-16T06:18:47.151 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T06:18:47.151 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-16T06:18:47.151 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T06:18:47.151 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T06:18:47.151 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-16T06:18:47.151 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-16T06:18:47.151 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-16T06:18:47.152 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:18:47.152 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:18:47.152 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:18:47.152 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.192403/ceph-mon.b.asok 2024-09-16T06:18:47.152 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T06:18:47.152 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.192403/ceph-mon.b.asok config get fsid 2024-09-16T06:18:47.229 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-09-16T06:18:47.229 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T06:18:47.230 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-16T06:18:47.230 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T06:18:47.230 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T06:18:47.230 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-16T06:18:47.230 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-16T06:18:47.230 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-16T06:18:47.230 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:18:47.230 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:18:47.230 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:18:47.230 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.192403/ceph-mon.b.asok 2024-09-16T06:18:47.230 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T06:18:47.230 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.192403/ceph-mon.b.asok config get mon_host 2024-09-16T06:18:47.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:142: TEST_check_version_health_2: kill_daemons td/ver-health KILL mds.m 2024-09-16T06:18:47.307 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:18:47.307 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:18:47.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:18:47.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:18:47.307 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:18:47.413 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:18:47.413 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:143: TEST_check_version_health_2: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-09-16T06:18:47.413 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:143: TEST_check_version_health_2: run_mds td/ver-health m 2024-09-16T06:18:47.413 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ver-health 2024-09-16T06:18:47.413 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-09-16T06:18:47.413 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=m 2024-09-16T06:18:47.414 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-09-16T06:18:47.414 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ver-health/m 2024-09-16T06:18:47.414 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-09-16T06:18:47.414 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:18:47.414 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:18:47.414 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:18:47.414 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:18:47.414 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:18:47.414 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:18:47.414 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id m --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ver-health/m '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' 2024-09-16T06:18:47.756 INFO:tasks.workunit.client.0.smithi178.stdout:starting mds.m at 2024-09-16T06:18:47.760 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:145: TEST_check_version_health_2: wait_for_health_string 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-09-16T06:18:47.760 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_WARN .*There are daemons running an older version of ceph' 2024-09-16T06:18:47.760 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2024-09-16T06:18:47.760 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2024-09-16T06:18:47.760 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2024-09-16T06:18:47.761 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2024-09-16T06:18:47.761 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T06:18:47.761 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T06:18:47.761 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-09-16T06:18:48.286 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T06:18:49.288 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T06:18:49.288 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T06:18:49.288 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T06:18:49.288 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-09-16T06:18:49.899 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T06:18:50.901 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T06:18:50.901 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T06:18:50.901 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T06:18:50.901 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-09-16T06:18:51.360 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T06:18:52.362 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T06:18:52.362 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T06:18:52.362 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T06:18:52.362 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-09-16T06:18:52.904 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T06:18:53.906 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T06:18:53.906 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T06:18:53.906 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T06:18:53.906 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-09-16T06:18:54.465 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T06:18:55.466 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T06:18:55.467 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T06:18:55.467 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T06:18:55.467 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-09-16T06:18:56.020 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2024-09-16T06:18:56.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2024-09-16T06:18:56.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2024-09-16T06:18:56.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2024-09-16T06:18:56.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2024-09-16T06:18:56.021 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:147: TEST_check_version_health_2: ceph health detail 2024-09-16T06:18:56.628 INFO:tasks.workunit.client.0.smithi178.stdout:HEALTH_WARN There are daemons running an older version of ceph 2024-09-16T06:18:56.628 INFO:tasks.workunit.client.0.smithi178.stdout:[WRN] DAEMON_OLD_VERSION: There are daemons running an older version of ceph 2024-09-16T06:18:56.629 INFO:tasks.workunit.client.0.smithi178.stdout: mon.b mds.m are running an older version of ceph: 01.00.00-gversion-test 2024-09-16T06:18:56.640 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:149: TEST_check_version_health_2: ceph health detail 2024-09-16T06:18:56.640 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:149: TEST_check_version_health_2: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-09-16T06:18:57.201 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:150: TEST_check_version_health_2: ceph health detail 2024-09-16T06:18:57.201 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:150: TEST_check_version_health_2: grep -q '^[[]WRN[]] DAEMON_OLD_VERSION: There are daemons running an older version of ceph' 2024-09-16T06:18:57.741 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:151: TEST_check_version_health_2: ceph health detail 2024-09-16T06:18:57.742 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:151: TEST_check_version_health_2: grep -q 'mon.b mds.m are running an older version of ceph: 01.00.00-gversion-test' 2024-09-16T06:18:58.299 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:153: TEST_check_version_health_2: kill_daemons td/ver-health KILL osd.2 2024-09-16T06:18:58.299 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:18:58.299 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:18:58.299 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:18:58.299 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:18:58.299 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:18:58.407 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:18:58.407 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:154: TEST_check_version_health_2: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-09-16T06:18:58.407 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:154: TEST_check_version_health_2: activate_osd td/ver-health 2 2024-09-16T06:18:58.407 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-09-16T06:18:58.407 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T06:18:58.407 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-16T06:18:58.407 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T06:18:58.407 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/2 2024-09-16T06:18:58.407 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T06:18:58.407 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:18:58.408 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:18:58.408 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:18:58.408 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/2' 2024-09-16T06:18:58.408 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2024-09-16T06:18:58.408 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T06:18:58.408 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T06:18:58.408 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-16T06:18:58.408 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T06:18:58.408 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:18:58.408 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:18:58.408 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:18:58.408 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:18:58.409 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:18:58.409 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:18:58.409 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:18:58.409 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:18:58.409 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-16T06:18:58.409 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-16T06:18:58.409 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:18:58.409 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:18:58.409 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:18:58.409 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:18:58.409 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T06:18:58.409 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T06:18:58.409 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/2 2024-09-16T06:18:58.409 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-16T06:18:58.410 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.2 2024-09-16T06:18:58.411 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T06:18:58.411 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/2/whoami 2024-09-16T06:18:58.411 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-16T06:18:58.411 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T06:18:58.411 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T06:18:58.411 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T06:18:58.432 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:18:58.431+0000 7f2a4f614640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:18:58.513 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:18:58.512+0000 7f2a4f614640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:18:58.514 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:18:58.513+0000 7f2a4f614640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:18:59.338 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-16T06:18:59.339 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:18:59.339 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T06:18:59.339 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:18:59.339 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:18:59.339 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:18:59.339 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:18:59.340 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:18:59.340 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:18:59.340 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:18:59.708 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:19:00.106 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:00.106+0000 7f2a4f614640 -1 Falling back to public interface 2024-09-16T06:19:00.710 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:19:00.710 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:19:00.710 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:19:00.710 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T06:19:00.711 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:19:00.711 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:19:01.086 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:19:02.087 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:19:02.088 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:19:02.088 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T06:19:02.088 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:19:02.088 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:19:02.089 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:19:02.456 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:19:03.215 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:03.215+0000 7f2a4f614640 -1 osd.2 17 log_to_monitors true 2024-09-16T06:19:03.457 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:19:03.458 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:19:03.458 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T06:19:03.458 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T06:19:03.458 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:19:03.458 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:19:04.290 INFO:tasks.workunit.client.0.smithi178.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 18 last_clean_interval [15,17) [v2:127.0.0.1:6816/4243869269,v1:127.0.0.1:6817/4243869269] [v2:127.0.0.1:6818/4243869269,v1:127.0.0.1:6819/4243869269] exists,up 27c1c00d-2693-48c1-b68b-6821d0e79563 2024-09-16T06:19:04.291 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:19:04.291 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:19:04.291 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:19:04.292 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:155: TEST_check_version_health_2: kill_daemons td/ver-health KILL osd.0 2024-09-16T06:19:04.292 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:19:04.292 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:19:04.292 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:19:04.292 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:19:04.292 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:19:04.399 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:19:04.399 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:156: TEST_check_version_health_2: ceph_debug_version_for_testing=02.00.00-gversion-test 2024-09-16T06:19:04.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:156: TEST_check_version_health_2: activate_osd td/ver-health 0 2024-09-16T06:19:04.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-09-16T06:19:04.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T06:19:04.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T06:19:04.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T06:19:04.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/0 2024-09-16T06:19:04.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T06:19:04.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:19:04.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:19:04.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:19:04.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/0' 2024-09-16T06:19:04.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2024-09-16T06:19:04.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T06:19:04.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T06:19:04.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-16T06:19:04.401 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T06:19:04.401 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:19:04.401 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:19:04.401 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:19:04.401 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:19:04.401 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:19:04.401 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:19:04.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:19:04.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:19:04.402 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-16T06:19:04.402 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-16T06:19:04.402 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:19:04.402 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:19:04.402 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:19:04.402 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:19:04.402 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T06:19:04.402 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T06:19:04.402 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/0 2024-09-16T06:19:04.402 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T06:19:04.403 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T06:19:04.403 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T06:19:04.403 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/0/whoami 2024-09-16T06:19:04.404 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T06:19:04.404 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T06:19:04.404 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T06:19:04.404 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T06:19:04.426 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:04.425+0000 7fe5816b4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:19:04.521 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:04.521+0000 7fe5816b4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:19:04.523 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:04.522+0000 7fe5816b4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:19:04.996 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T06:19:04.996 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:19:04.996 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T06:19:04.996 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:19:04.996 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:19:04.996 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:19:04.996 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:19:04.997 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:19:04.997 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:19:04.997 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:19:05.481 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:19:05.872 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:05.872+0000 7fe5816b4640 -1 Falling back to public interface 2024-09-16T06:19:06.482 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:19:06.483 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:19:06.483 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:19:06.483 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T06:19:06.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:19:06.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:19:06.993 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:19:07.994 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:19:07.994 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:19:07.995 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:19:07.995 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T06:19:07.995 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:19:07.995 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:19:08.377 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:19:09.378 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:19:09.378 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:19:09.378 INFO:tasks.workunit.client.0.smithi178.stdout:3 2024-09-16T06:19:09.379 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T06:19:09.379 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:19:09.379 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:19:09.547 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:09.547+0000 7fe5816b4640 -1 osd.0 20 log_to_monitors true 2024-09-16T06:19:10.398 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 23 up_thru 0 down_at 21 last_clean_interval [5,20) [v2:127.0.0.1:6800/1605722152,v1:127.0.0.1:6801/1605722152] [v2:127.0.0.1:6802/1605722152,v1:127.0.0.1:6803/1605722152] exists,up fc103cfa-4066-4971-96da-7da607e2104f 2024-09-16T06:19:10.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:19:10.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:19:10.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:19:10.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:158: TEST_check_version_health_2: wait_for_health_string 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-16T06:19:10.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-16T06:19:10.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2024-09-16T06:19:10.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2024-09-16T06:19:10.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2024-09-16T06:19:10.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2024-09-16T06:19:10.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T06:19:10.400 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T06:19:10.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-16T06:19:10.934 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T06:19:11.937 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T06:19:11.937 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T06:19:11.937 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T06:19:11.937 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-16T06:19:12.528 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T06:19:13.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T06:19:13.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T06:19:13.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T06:19:13.530 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-16T06:19:14.066 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T06:19:15.068 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T06:19:15.068 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T06:19:15.068 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T06:19:15.069 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-16T06:19:15.531 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2024-09-16T06:19:15.531 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2024-09-16T06:19:15.531 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2024-09-16T06:19:15.531 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2024-09-16T06:19:15.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2024-09-16T06:19:15.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:160: TEST_check_version_health_2: ceph health detail 2024-09-16T06:19:16.086 INFO:tasks.workunit.client.0.smithi178.stdout:HEALTH_ERR There are daemons running multiple old versions of ceph 2024-09-16T06:19:16.087 INFO:tasks.workunit.client.0.smithi178.stdout:[ERR] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph 2024-09-16T06:19:16.087 INFO:tasks.workunit.client.0.smithi178.stdout: mon.b osd.2 mds.m are running an older version of ceph: 01.00.00-gversion-test 2024-09-16T06:19:16.087 INFO:tasks.workunit.client.0.smithi178.stdout: osd.0 is running an older version of ceph: 02.00.00-gversion-test 2024-09-16T06:19:16.098 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:161: TEST_check_version_health_2: ceph health 2024-09-16T06:19:16.098 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:161: TEST_check_version_health_2: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-16T06:19:16.598 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:162: TEST_check_version_health_2: ceph health detail 2024-09-16T06:19:16.598 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:162: TEST_check_version_health_2: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-16T06:19:17.147 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:163: TEST_check_version_health_2: ceph health detail 2024-09-16T06:19:17.147 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:163: TEST_check_version_health_2: grep -q '^[[]ERR[]] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph' 2024-09-16T06:19:17.768 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:164: TEST_check_version_health_2: ceph health detail 2024-09-16T06:19:17.768 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:164: TEST_check_version_health_2: grep -q 'mon.b osd.2 mds.m are running an older version of ceph: 01.00.00-gversion-test' 2024-09-16T06:19:18.320 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:165: TEST_check_version_health_2: ceph health detail 2024-09-16T06:19:18.320 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:165: TEST_check_version_health_2: grep -q 'osd.0 is running an older version of ceph: 02.00.00-gversion-test' 2024-09-16T06:19:18.851 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:35: run: teardown td/ver-health 2024-09-16T06:19:18.851 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-16T06:19:18.852 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T06:19:18.852 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-16T06:19:18.852 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:19:18.852 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:19:18.852 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:19:18.852 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:19:18.852 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:19:18.962 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:19:18.962 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:19:18.963 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:19:18.963 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:19:18.964 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:19:18.964 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:19:18.964 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:19:18.965 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:19:18.965 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:19:18.966 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:19:18.966 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:19:18.966 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:19:18.967 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T06:19:18.967 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-16T06:19:18.984 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:19:18.984 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:19:18.984 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:19:18.984 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.192403 2024-09-16T06:19:18.986 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:19:18.986 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:19:18.986 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:32: run: for func in $funcs 2024-09-16T06:19:18.986 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:33: run: setup td/ver-health 2024-09-16T06:19:18.986 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2024-09-16T06:19:18.986 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2024-09-16T06:19:18.986 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-16T06:19:18.987 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T06:19:18.987 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-16T06:19:18.987 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:19:18.987 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:19:18.987 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:19:18.987 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:19:18.987 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:19:18.987 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:19:18.988 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:19:18.988 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:19:18.988 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:19:18.990 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:19:18.990 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:19:18.990 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:19:18.990 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:19:18.990 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:19:18.991 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:19:18.991 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:19:18.991 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:19:18.993 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T06:19:18.993 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-16T06:19:18.994 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:19:18.994 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:19:18.994 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:19:18.994 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.192403 2024-09-16T06:19:18.995 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:19:18.995 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:19:18.995 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2024-09-16T06:19:18.996 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T06:19:18.996 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:19:18.996 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:19:18.996 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.192403 2024-09-16T06:19:18.999 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T06:19:18.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T06:19:18.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T06:19:18.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2024-09-16T06:19:18.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:34: run: TEST_check_version_health_3 td/ver-health 2024-09-16T06:19:18.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:170: TEST_check_version_health_3: local dir=td/ver-health 2024-09-16T06:19:18.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:173: TEST_check_version_health_3: CEPH_ARGS='--fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T06:19:18.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:175: TEST_check_version_health_3: setup td/ver-health 2024-09-16T06:19:18.999 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2024-09-16T06:19:19.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2024-09-16T06:19:19.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-16T06:19:19.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T06:19:19.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-16T06:19:19.000 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:19:19.000 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:19:19.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:19:19.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:19:19.000 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:19:19.001 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:19:19.001 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:19:19.002 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:19:19.002 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:19:19.002 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:19:19.003 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:19:19.003 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:19:19.004 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:19:19.004 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:19:19.004 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:19:19.004 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:19:19.004 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:19:19.005 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T06:19:19.005 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-16T06:19:19.006 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:19:19.006 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:19:19.006 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:19:19.006 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.192403 2024-09-16T06:19:19.007 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:19:19.007 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:19:19.007 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2024-09-16T06:19:19.008 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T06:19:19.008 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:19:19.008 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:19:19.009 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.192403 2024-09-16T06:19:19.010 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T06:19:19.011 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T06:19:19.011 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T06:19:19.011 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2024-09-16T06:19:19.011 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:178: TEST_check_version_health_3: run_mon td/ver-health a --public-addr=127.0.0.1:7165 2024-09-16T06:19:19.011 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-09-16T06:19:19.011 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T06:19:19.011 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T06:19:19.011 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T06:19:19.011 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/a 2024-09-16T06:19:19.011 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ver-health/a --run-dir=td/ver-health --public-addr=127.0.0.1:7165 2024-09-16T06:19:19.393 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T06:19:19.393 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:19:19.394 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:19:19.394 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:19:19.394 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:19:19.394 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:19:19.394 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:19:19.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/a '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7165 2024-09-16T06:19:19.584 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T06:19:19.586 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T06:19:19.586 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T06:19:19.586 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T06:19:19.586 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T06:19:19.586 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T06:19:19.586 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T06:19:19.586 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T06:19:19.586 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T06:19:19.587 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:19:19.587 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:19:19.587 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:19:19.587 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.192403/ceph-mon.a.asok 2024-09-16T06:19:19.587 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T06:19:19.587 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.192403/ceph-mon.a.asok config get fsid 2024-09-16T06:19:19.664 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T06:19:19.664 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T06:19:19.664 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T06:19:19.664 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T06:19:19.664 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T06:19:19.665 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T06:19:19.665 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T06:19:19.665 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T06:19:19.665 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:19:19.665 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:19:19.665 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:19:19.665 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.192403/ceph-mon.a.asok 2024-09-16T06:19:19.665 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T06:19:19.665 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.192403/ceph-mon.a.asok config get mon_host 2024-09-16T06:19:19.743 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:179: TEST_check_version_health_3: run_mon td/ver-health b --public-addr=127.0.0.1:7166 2024-09-16T06:19:19.743 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-09-16T06:19:19.743 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T06:19:19.743 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-09-16T06:19:19.743 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T06:19:19.743 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/b 2024-09-16T06:19:19.743 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ver-health/b --run-dir=td/ver-health --public-addr=127.0.0.1:7166 2024-09-16T06:19:20.400 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T06:19:20.400 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:19:20.400 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:19:20.400 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:19:20.400 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:19:20.401 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:19:20.401 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:19:20.401 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/b '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7166 2024-09-16T06:19:20.691 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T06:19:20.692 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-09-16T06:19:20.692 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T06:19:20.692 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-16T06:19:20.692 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T06:19:20.692 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T06:19:20.692 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-16T06:19:20.692 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-16T06:19:20.692 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-16T06:19:20.693 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:19:20.693 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:19:20.693 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:19:20.694 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.192403/ceph-mon.b.asok 2024-09-16T06:19:20.694 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T06:19:20.694 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.192403/ceph-mon.b.asok config get fsid 2024-09-16T06:19:20.774 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-09-16T06:19:20.774 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T06:19:20.774 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-16T06:19:20.774 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T06:19:20.774 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T06:19:20.774 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-16T06:19:20.774 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-16T06:19:20.774 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-16T06:19:20.775 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T06:19:20.775 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:19:20.775 INFO:tasks.workunit.client.0.smithi178.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:19:20.775 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.192403/ceph-mon.b.asok 2024-09-16T06:19:20.775 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T06:19:20.776 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.192403/ceph-mon.b.asok config get mon_host 2024-09-16T06:19:20.856 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:181: TEST_check_version_health_3: local start_osd_time=301 2024-09-16T06:19:20.856 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:183: TEST_check_version_health_3: EXTRA_OPTS=' --osd-objectstore=memstore' 2024-09-16T06:19:20.856 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:183: TEST_check_version_health_3: run_osd td/ver-health 0 2024-09-16T06:19:20.856 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-09-16T06:19:20.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T06:19:20.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T06:19:20.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T06:19:20.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/0 2024-09-16T06:19:20.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T06:19:20.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:19:20.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:19:20.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:19:20.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/0' 2024-09-16T06:19:20.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2024-09-16T06:19:20.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T06:19:20.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-objectstore=memstore' 2024-09-16T06:19:20.857 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-16T06:19:20.857 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T06:19:20.858 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:19:20.858 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:19:20.858 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:19:20.858 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:19:20.858 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:19:20.858 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:19:20.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:19:20.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:19:20.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T06:19:20.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T06:19:20.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-16T06:19:20.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-16T06:19:20.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:19:20.858 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:19:20.859 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:19:20.859 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:19:20.859 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T06:19:20.859 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T06:19:20.859 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/0 2024-09-16T06:19:20.859 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T06:19:20.859 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a75b435f-913e-418c-9121-ad65d767e219 2024-09-16T06:19:20.859 INFO:tasks.workunit.client.0.smithi178.stdout:add osd0 a75b435f-913e-418c-9121-ad65d767e219 2024-09-16T06:19:20.860 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a75b435f-913e-418c-9121-ad65d767e219' 2024-09-16T06:19:20.860 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T06:19:20.873 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDozedmoDUJNBAA48XJElfDynzLJzy8rqzP0w== 2024-09-16T06:19:20.873 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDozedmoDUJNBAA48XJElfDynzLJzy8rqzP0w=="}' 2024-09-16T06:19:20.873 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a75b435f-913e-418c-9121-ad65d767e219 -i td/ver-health/0/new.json 2024-09-16T06:19:27.190 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:19:27.200 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/0/new.json 2024-09-16T06:19:27.201 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDozedmoDUJNBAA48XJElfDynzLJzy8rqzP0w== --osd-uuid a75b435f-913e-418c-9121-ad65d767e219 2024-09-16T06:19:27.230 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:27.230+0000 7f910a7a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:19:27.232 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:27.232+0000 7f910a7a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:19:27.233 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:27.233+0000 7f910a7a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:19:27.349 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:27.349+0000 7f910a7a8640 -1 memstore(td/ver-health/0) td/ver-health/0 2024-09-16T06:19:27.883 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/0/keyring 2024-09-16T06:19:27.883 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T06:19:27.884 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd0 key to auth repository 2024-09-16T06:19:27.884 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T06:19:27.884 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T06:19:28.081 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T06:19:28.082 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T06:19:28.082 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T06:19:28.082 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T06:19:28.083 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T06:19:28.083 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T06:19:28.112 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:28.108+0000 7f209b832640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:19:28.112 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:28.111+0000 7f209b832640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:19:28.113 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:28.112+0000 7f209b832640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:19:28.114 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:28.114+0000 7f209b832640 -1 Falling back to public interface 2024-09-16T06:19:28.123 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:28.122+0000 7f209b832640 -1 osd.0 0 log_to_monitors true 2024-09-16T06:19:28.317 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T06:19:28.317 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:19:28.317 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T06:19:28.317 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:19:28.317 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:19:28.317 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:19:28.318 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:19:28.318 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:19:28.318 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:19:28.318 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:19:28.632 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:19:29.633 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:19:29.633 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:19:29.634 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:19:29.634 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T06:19:29.634 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:19:29.635 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:19:29.856 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1045529167,v1:127.0.0.1:6801/1045529167] [v2:127.0.0.1:6802/1045529167,v1:127.0.0.1:6803/1045529167] exists,up a75b435f-913e-418c-9121-ad65d767e219 2024-09-16T06:19:29.860 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:19:29.860 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:19:29.860 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:19:29.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:184: TEST_check_version_health_3: EXTRA_OPTS=' --osd-objectstore=memstore' 2024-09-16T06:19:29.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:184: TEST_check_version_health_3: run_osd td/ver-health 1 2024-09-16T06:19:29.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-09-16T06:19:29.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T06:19:29.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T06:19:29.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T06:19:29.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/1 2024-09-16T06:19:29.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T06:19:29.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:19:29.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:19:29.861 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:19:29.862 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/1' 2024-09-16T06:19:29.862 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2024-09-16T06:19:29.862 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T06:19:29.862 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-objectstore=memstore' 2024-09-16T06:19:29.862 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-16T06:19:29.862 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T06:19:29.862 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:19:29.862 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:19:29.862 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:19:29.862 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:19:29.862 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:19:29.862 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:19:29.863 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:19:29.863 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:19:29.863 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T06:19:29.863 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T06:19:29.863 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-16T06:19:29.863 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-16T06:19:29.863 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:19:29.863 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:19:29.863 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:19:29.863 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:19:29.864 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T06:19:29.864 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T06:19:29.864 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/1 2024-09-16T06:19:29.864 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T06:19:29.864 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6a30de64-dcd2-46d8-b230-35823b797628 2024-09-16T06:19:29.864 INFO:tasks.workunit.client.0.smithi178.stdout:add osd1 6a30de64-dcd2-46d8-b230-35823b797628 2024-09-16T06:19:29.865 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 6a30de64-dcd2-46d8-b230-35823b797628' 2024-09-16T06:19:29.865 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T06:19:29.878 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDxzedmkd1ONBAAVshC5e0XzyVgk9/IPTS1Ng== 2024-09-16T06:19:29.878 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDxzedmkd1ONBAAVshC5e0XzyVgk9/IPTS1Ng=="}' 2024-09-16T06:19:29.878 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6a30de64-dcd2-46d8-b230-35823b797628 -i td/ver-health/1/new.json 2024-09-16T06:19:30.266 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:19:30.276 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/1/new.json 2024-09-16T06:19:30.276 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDxzedmkd1ONBAAVshC5e0XzyVgk9/IPTS1Ng== --osd-uuid 6a30de64-dcd2-46d8-b230-35823b797628 2024-09-16T06:19:30.305 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:30.304+0000 7f962301c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:19:30.306 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:30.306+0000 7f962301c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:19:30.308 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:30.307+0000 7f962301c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:19:30.417 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:30.416+0000 7f962301c640 -1 memstore(td/ver-health/1) td/ver-health/1 2024-09-16T06:19:31.059 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/1/keyring 2024-09-16T06:19:31.059 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T06:19:31.060 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd1 key to auth repository 2024-09-16T06:19:31.060 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T06:19:31.060 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T06:19:31.261 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.1 2024-09-16T06:19:31.262 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T06:19:31.262 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T06:19:31.262 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T06:19:31.262 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T06:19:31.263 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T06:19:31.288 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:31.288+0000 7f3e73f45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:19:31.348 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:31.348+0000 7f3e73f45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:19:31.349 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:31.349+0000 7f3e73f45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:19:31.351 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:31.350+0000 7f3e73f45640 -1 Falling back to public interface 2024-09-16T06:19:31.360 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:31.360+0000 7f3e73f45640 -1 osd.1 0 log_to_monitors true 2024-09-16T06:19:31.751 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T06:19:31.751 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:19:31.751 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T06:19:31.751 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:19:31.751 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:19:31.752 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:19:31.752 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:19:31.752 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:19:31.752 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:19:31.752 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:19:32.479 INFO:tasks.workunit.client.0.smithi178.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/990932774,v1:127.0.0.1:6809/990932774] [v2:127.0.0.1:6810/990932774,v1:127.0.0.1:6811/990932774] exists,up 6a30de64-dcd2-46d8-b230-35823b797628 2024-09-16T06:19:32.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:19:32.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:19:32.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:19:32.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:185: TEST_check_version_health_3: EXTRA_OPTS=' --osd-objectstore=memstore' 2024-09-16T06:19:32.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:185: TEST_check_version_health_3: run_osd td/ver-health 2 2024-09-16T06:19:32.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-09-16T06:19:32.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T06:19:32.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T06:19:32.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T06:19:32.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/2 2024-09-16T06:19:32.484 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T06:19:32.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:19:32.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:19:32.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:19:32.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/2' 2024-09-16T06:19:32.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2024-09-16T06:19:32.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T06:19:32.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-objectstore=memstore' 2024-09-16T06:19:32.485 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-16T06:19:32.485 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T06:19:32.485 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:19:32.485 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:19:32.485 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:19:32.485 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:19:32.486 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:19:32.486 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:19:32.486 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:19:32.486 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:19:32.486 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T06:19:32.486 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T06:19:32.486 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-16T06:19:32.486 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-16T06:19:32.486 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:19:32.486 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:19:32.486 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:19:32.486 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:19:32.486 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T06:19:32.487 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T06:19:32.487 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/2 2024-09-16T06:19:32.487 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T06:19:32.487 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=df016447-1ff0-4d75-8793-ac631a4cd55c 2024-09-16T06:19:32.487 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 df016447-1ff0-4d75-8793-ac631a4cd55c' 2024-09-16T06:19:32.487 INFO:tasks.workunit.client.0.smithi178.stdout:add osd2 df016447-1ff0-4d75-8793-ac631a4cd55c 2024-09-16T06:19:32.487 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T06:19:32.501 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD0zedmUV7YHRAAtnpjuxU05hfESOFplXLbDA== 2024-09-16T06:19:32.501 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD0zedmUV7YHRAAtnpjuxU05hfESOFplXLbDA=="}' 2024-09-16T06:19:32.501 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new df016447-1ff0-4d75-8793-ac631a4cd55c -i td/ver-health/2/new.json 2024-09-16T06:19:33.215 INFO:tasks.workunit.client.0.smithi178.stdout:2 2024-09-16T06:19:33.224 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/2/new.json 2024-09-16T06:19:33.225 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD0zedmUV7YHRAAtnpjuxU05hfESOFplXLbDA== --osd-uuid df016447-1ff0-4d75-8793-ac631a4cd55c 2024-09-16T06:19:33.254 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:33.254+0000 7f4b451fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:19:33.342 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:33.341+0000 7f4b451fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:19:33.343 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:33.342+0000 7f4b451fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:19:33.415 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:33.415+0000 7f4b451fd640 -1 memstore(td/ver-health/2) td/ver-health/2 2024-09-16T06:19:34.109 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/2/keyring 2024-09-16T06:19:34.109 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T06:19:34.110 INFO:tasks.workunit.client.0.smithi178.stdout:adding osd2 key to auth repository 2024-09-16T06:19:34.110 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T06:19:34.110 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T06:19:34.313 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.2 2024-09-16T06:19:34.313 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T06:19:34.313 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T06:19:34.314 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T06:19:34.314 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T06:19:34.314 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T06:19:34.340 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:34.339+0000 7f4538214640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:19:34.346 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:34.345+0000 7f4538214640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:19:34.347 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:34.346+0000 7f4538214640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:19:34.348 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:34.348+0000 7f4538214640 -1 Falling back to public interface 2024-09-16T06:19:34.359 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:34.358+0000 7f4538214640 -1 osd.2 0 log_to_monitors true 2024-09-16T06:19:34.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T06:19:34.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:19:34.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T06:19:34.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:19:34.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:19:34.532 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:19:34.533 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:19:34.533 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:19:34.533 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:19:34.533 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:19:35.026 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:19:36.028 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:19:36.028 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:19:36.028 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:19:36.029 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T06:19:36.029 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:19:36.029 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:19:36.220 INFO:tasks.workunit.client.0.smithi178.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/1359854828,v1:127.0.0.1:6817/1359854828] [v2:127.0.0.1:6818/1359854828,v1:127.0.0.1:6819/1359854828] exists,up df016447-1ff0-4d75-8793-ac631a4cd55c 2024-09-16T06:19:36.220 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:19:36.221 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:19:36.221 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:19:36.221 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:187: TEST_check_version_health_3: local warn_older_version_delay=36 2024-09-16T06:19:36.221 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:189: TEST_check_version_health_3: sleep 5 2024-09-16T06:19:41.221 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:190: TEST_check_version_health_3: ceph health detail 2024-09-16T06:19:41.409 INFO:tasks.workunit.client.0.smithi178.stdout:HEALTH_OK 2024-09-16T06:19:41.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:192: TEST_check_version_health_3: ceph health detail 2024-09-16T06:19:41.419 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:192: TEST_check_version_health_3: grep DAEMON_OLD_VERSION 2024-09-16T06:19:41.634 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:193: TEST_check_version_health_3: ceph tell 'mon.*' injectargs '--mon_warn_older_version_delay 36' 2024-09-16T06:19:41.769 INFO:tasks.workunit.client.0.smithi178.stdout:mon.a: {} 2024-09-16T06:19:41.800 INFO:tasks.workunit.client.0.smithi178.stdout:mon.b: {} 2024-09-16T06:19:41.806 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:194: TEST_check_version_health_3: kill_daemons td/ver-health KILL osd.1 2024-09-16T06:19:41.807 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:19:41.807 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:19:41.807 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:19:41.807 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:19:41.807 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:19:41.915 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:19:41.915 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:195: TEST_check_version_health_3: EXTRA_OPTS=' --osd-objectstore=memstore' 2024-09-16T06:19:41.915 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:195: TEST_check_version_health_3: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-09-16T06:19:41.915 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:195: TEST_check_version_health_3: activate_osd td/ver-health 1 2024-09-16T06:19:41.916 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-09-16T06:19:41.916 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T06:19:41.916 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-16T06:19:41.916 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T06:19:41.916 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/1 2024-09-16T06:19:41.916 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T06:19:41.916 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:19:41.916 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:19:41.916 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:19:41.916 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/1' 2024-09-16T06:19:41.916 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2024-09-16T06:19:41.916 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T06:19:41.916 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-objectstore=memstore' 2024-09-16T06:19:41.917 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-16T06:19:41.917 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T06:19:41.917 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:19:41.917 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:19:41.917 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:19:41.917 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:19:41.917 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:19:41.917 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:19:41.917 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:19:41.917 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:19:41.917 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-16T06:19:41.917 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-16T06:19:41.917 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:19:41.917 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:19:41.918 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:19:41.918 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:19:41.918 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T06:19:41.918 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T06:19:41.918 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/1 2024-09-16T06:19:41.918 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.1 2024-09-16T06:19:41.919 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-16T06:19:41.919 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T06:19:41.919 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/1/whoami 2024-09-16T06:19:41.919 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-16T06:19:41.919 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T06:19:41.919 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T06:19:41.919 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T06:19:41.941 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:41.941+0000 7f7b770b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:19:42.003 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:42.002+0000 7f7b770b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:19:42.004 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:42.004+0000 7f7b770b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:19:42.008 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:42.005+0000 7f7b770b6640 -1 Falling back to public interface 2024-09-16T06:19:42.015 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:19:42.015+0000 7f7b770b6640 -1 osd.1 0 log_to_monitors true 2024-09-16T06:19:42.453 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-16T06:19:42.453 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:19:42.453 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T06:19:42.453 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:19:42.453 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:19:42.454 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:19:42.454 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:19:42.454 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:19:42.454 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:19:42.454 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:19:42.701 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:19:43.702 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:19:43.702 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:19:43.703 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:19:43.703 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T06:19:43.703 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:19:43.703 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T06:19:43.887 INFO:tasks.workunit.client.0.smithi178.stdout:osd.1 up in weight 1 up_from 19 up_thru 0 down_at 17 last_clean_interval [0,0) [v2:127.0.0.1:6808/1569806422,v1:127.0.0.1:6809/1569806422] [v2:127.0.0.1:6810/1569806422,v1:127.0.0.1:6811/1569806422] exists,up 6a30de64-dcd2-46d8-b230-35823b797628 2024-09-16T06:19:43.887 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:19:43.887 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:19:43.887 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:19:43.887 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:200: TEST_check_version_health_3: sleep 10 2024-09-16T06:19:53.889 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:202: TEST_check_version_health_3: ceph health detail 2024-09-16T06:19:53.889 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:202: TEST_check_version_health_3: grep DAEMON_OLD_VERSION 2024-09-16T06:19:54.088 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:205: TEST_check_version_health_3: wait_for_health_string 'HEALTH_WARN .*There is a daemon running an older version of ceph' 20 2024-09-16T06:19:54.088 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T06:19:54.088 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2024-09-16T06:19:54.088 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2024-09-16T06:19:54.088 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2024-09-16T06:19:54.088 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2024-09-16T06:19:54.089 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T06:19:54.089 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T06:19:54.089 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T06:19:54.275 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T06:19:55.277 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T06:19:55.277 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T06:19:55.277 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T06:19:55.277 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T06:19:55.473 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T06:19:56.475 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T06:19:56.475 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T06:19:56.475 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T06:19:56.475 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T06:19:56.665 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T06:19:57.667 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T06:19:57.667 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T06:19:57.667 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T06:19:57.667 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T06:19:57.863 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T06:19:58.864 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T06:19:58.864 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T06:19:58.865 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T06:19:58.865 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T06:19:59.056 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T06:20:00.057 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T06:20:00.058 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T06:20:00.058 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T06:20:00.058 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T06:20:00.331 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T06:20:01.333 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T06:20:01.333 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T06:20:01.333 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T06:20:01.333 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T06:20:01.528 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T06:20:02.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T06:20:02.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T06:20:02.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T06:20:02.529 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T06:20:02.722 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T06:20:03.723 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T06:20:03.723 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T06:20:03.724 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T06:20:03.724 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T06:20:03.922 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T06:20:04.923 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T06:20:04.924 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T06:20:04.924 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T06:20:04.924 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T06:20:05.121 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2024-09-16T06:20:05.121 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2024-09-16T06:20:05.121 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2024-09-16T06:20:05.121 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2024-09-16T06:20:05.122 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2024-09-16T06:20:05.122 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:207: TEST_check_version_health_3: ceph health detail 2024-09-16T06:20:05.367 INFO:tasks.workunit.client.0.smithi178.stdout:HEALTH_WARN There is a daemon running an older version of ceph 2024-09-16T06:20:05.367 INFO:tasks.workunit.client.0.smithi178.stdout:[WRN] DAEMON_OLD_VERSION: There is a daemon running an older version of ceph 2024-09-16T06:20:05.368 INFO:tasks.workunit.client.0.smithi178.stdout: osd.1 is running an older version of ceph: 01.00.00-gversion-test 2024-09-16T06:20:05.377 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:209: TEST_check_version_health_3: ceph health detail 2024-09-16T06:20:05.377 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:209: TEST_check_version_health_3: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T06:20:05.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:210: TEST_check_version_health_3: ceph health detail 2024-09-16T06:20:05.592 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:210: TEST_check_version_health_3: grep -q '^[[]WRN[]] DAEMON_OLD_VERSION: There is a daemon running an older version of ceph' 2024-09-16T06:20:05.864 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:211: TEST_check_version_health_3: ceph health detail 2024-09-16T06:20:05.864 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:211: TEST_check_version_health_3: grep -q 'osd.1 is running an older version of ceph: 01.00.00-gversion-test' 2024-09-16T06:20:06.089 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:213: TEST_check_version_health_3: kill_daemons td/ver-health KILL osd.2 2024-09-16T06:20:06.090 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:20:06.090 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:20:06.090 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:20:06.090 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:20:06.090 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:20:06.197 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:20:06.198 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:214: TEST_check_version_health_3: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-09-16T06:20:06.198 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:214: TEST_check_version_health_3: activate_osd td/ver-health 2 2024-09-16T06:20:06.198 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-09-16T06:20:06.198 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T06:20:06.198 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-16T06:20:06.198 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T06:20:06.198 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/2 2024-09-16T06:20:06.198 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T06:20:06.198 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:20:06.198 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:20:06.198 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:20:06.199 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/2' 2024-09-16T06:20:06.199 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2024-09-16T06:20:06.199 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T06:20:06.199 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T06:20:06.199 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-16T06:20:06.199 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T06:20:06.199 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:20:06.199 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:20:06.199 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:20:06.199 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:20:06.199 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:20:06.199 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:20:06.199 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:20:06.200 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:20:06.200 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-16T06:20:06.200 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-16T06:20:06.200 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:20:06.200 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:20:06.200 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:20:06.200 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:20:06.200 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T06:20:06.200 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T06:20:06.200 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/2 2024-09-16T06:20:06.200 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.2 2024-09-16T06:20:06.201 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-16T06:20:06.201 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T06:20:06.201 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/2/whoami 2024-09-16T06:20:06.202 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-16T06:20:06.202 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T06:20:06.202 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T06:20:06.202 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T06:20:06.223 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:20:06.222+0000 7fd8e1820640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:20:06.274 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:20:06.272+0000 7fd8e1820640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:20:06.275 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:20:06.273+0000 7fd8e1820640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:20:06.277 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:20:06.275+0000 7fd8e1820640 -1 Falling back to public interface 2024-09-16T06:20:06.291 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:20:06.290+0000 7fd8e1820640 -1 osd.2 0 log_to_monitors true 2024-09-16T06:20:06.462 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-16T06:20:06.463 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:20:06.463 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T06:20:06.463 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:20:06.463 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:20:06.463 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:20:06.463 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:20:06.464 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:20:06.464 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:20:06.464 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T06:20:07.104 INFO:tasks.workunit.client.0.smithi178.stdout:osd.2 up in weight 1 up_from 23 up_thru 0 down_at 21 last_clean_interval [0,0) [v2:127.0.0.1:6816/2955720826,v1:127.0.0.1:6817/2955720826] [v2:127.0.0.1:6818/2955720826,v1:127.0.0.1:6819/2955720826] exists,up df016447-1ff0-4d75-8793-ac631a4cd55c 2024-09-16T06:20:07.105 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:20:07.105 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:20:07.106 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:20:07.106 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:215: TEST_check_version_health_3: kill_daemons td/ver-health KILL osd.0 2024-09-16T06:20:07.106 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:20:07.106 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:20:07.106 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:20:07.106 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:20:07.106 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:20:07.213 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:20:07.214 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:216: TEST_check_version_health_3: ceph_debug_version_for_testing=02.00.00-gversion-test 2024-09-16T06:20:07.214 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:216: TEST_check_version_health_3: activate_osd td/ver-health 0 2024-09-16T06:20:07.214 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-09-16T06:20:07.214 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T06:20:07.214 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T06:20:07.214 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T06:20:07.214 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/0 2024-09-16T06:20:07.214 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T06:20:07.214 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T06:20:07.214 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T06:20:07.214 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T06:20:07.214 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/0' 2024-09-16T06:20:07.214 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2024-09-16T06:20:07.215 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T06:20:07.215 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T06:20:07.215 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-16T06:20:07.215 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T06:20:07.215 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T06:20:07.215 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T06:20:07.215 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T06:20:07.215 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:20:07.215 INFO:tasks.workunit.client.0.smithi178.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:20:07.215 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:20:07.215 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' 2024-09-16T06:20:07.215 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T06:20:07.215 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-16T06:20:07.215 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-16T06:20:07.216 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T06:20:07.216 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T06:20:07.216 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T06:20:07.216 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T06:20:07.216 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T06:20:07.216 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T06:20:07.216 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/0 2024-09-16T06:20:07.216 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T06:20:07.216 INFO:tasks.workunit.client.0.smithi178.stdout:start osd.0 2024-09-16T06:20:07.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=ac713091-da47-494b-b07f-e2979a1b2d0b --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.192403/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T06:20:07.217 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/0/whoami 2024-09-16T06:20:07.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T06:20:07.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T06:20:07.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T06:20:07.217 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T06:20:07.239 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:20:07.238+0000 7fa80a745640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:20:07.254 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:20:07.252+0000 7fa80a745640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:20:07.255 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:20:07.253+0000 7fa80a745640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T06:20:07.256 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:20:07.255+0000 7fa80a745640 -1 Falling back to public interface 2024-09-16T06:20:07.265 INFO:tasks.workunit.client.0.smithi178.stderr:2024-09-16T06:20:07.264+0000 7fa80a745640 -1 osd.0 0 log_to_monitors true 2024-09-16T06:20:07.450 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T06:20:07.451 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T06:20:07.451 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T06:20:07.451 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T06:20:07.451 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T06:20:07.451 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:20:07.451 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T06:20:07.451 INFO:tasks.workunit.client.0.smithi178.stdout:0 2024-09-16T06:20:07.451 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:20:07.452 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:20:07.786 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T06:20:08.787 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T06:20:08.788 INFO:tasks.workunit.client.0.smithi178.stdout:1 2024-09-16T06:20:08.788 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T06:20:08.788 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T06:20:08.788 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T06:20:08.788 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T06:20:08.969 INFO:tasks.workunit.client.0.smithi178.stdout:osd.0 up in weight 1 up_from 26 up_thru 0 down_at 24 last_clean_interval [0,0) [v2:127.0.0.1:6800/2195490132,v1:127.0.0.1:6801/2195490132] [v2:127.0.0.1:6802/2195490132,v1:127.0.0.1:6803/2195490132] exists,up a75b435f-913e-418c-9121-ad65d767e219 2024-09-16T06:20:08.971 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T06:20:08.971 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T06:20:08.971 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T06:20:08.971 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:218: TEST_check_version_health_3: wait_for_health_string 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-16T06:20:08.971 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-16T06:20:08.971 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2024-09-16T06:20:08.971 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2024-09-16T06:20:08.971 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2024-09-16T06:20:08.971 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2024-09-16T06:20:08.971 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T06:20:08.971 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T06:20:08.972 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-16T06:20:09.196 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T06:20:10.198 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T06:20:10.198 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T06:20:10.198 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T06:20:10.198 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-16T06:20:10.394 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2024-09-16T06:20:10.394 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2024-09-16T06:20:10.394 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2024-09-16T06:20:10.394 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2024-09-16T06:20:10.394 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2024-09-16T06:20:10.395 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:220: TEST_check_version_health_3: ceph health detail 2024-09-16T06:20:10.570 INFO:tasks.workunit.client.0.smithi178.stdout:HEALTH_ERR There are daemons running multiple old versions of ceph 2024-09-16T06:20:10.571 INFO:tasks.workunit.client.0.smithi178.stdout:[ERR] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph 2024-09-16T06:20:10.571 INFO:tasks.workunit.client.0.smithi178.stdout: osd.1 osd.2 are running an older version of ceph: 01.00.00-gversion-test 2024-09-16T06:20:10.571 INFO:tasks.workunit.client.0.smithi178.stdout: osd.0 is running an older version of ceph: 02.00.00-gversion-test 2024-09-16T06:20:10.580 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:221: TEST_check_version_health_3: ceph health detail 2024-09-16T06:20:10.580 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:221: TEST_check_version_health_3: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-16T06:20:10.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:222: TEST_check_version_health_3: ceph health detail 2024-09-16T06:20:10.881 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:222: TEST_check_version_health_3: grep -q '^[[]ERR[]] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph' 2024-09-16T06:20:11.130 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:223: TEST_check_version_health_3: ceph health detail 2024-09-16T06:20:11.130 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:223: TEST_check_version_health_3: grep -q 'osd.1 osd.2 are running an older version of ceph: 01.00.00-gversion-test' 2024-09-16T06:20:11.343 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:224: TEST_check_version_health_3: ceph health detail 2024-09-16T06:20:11.344 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:224: TEST_check_version_health_3: grep -q 'osd.0 is running an older version of ceph: 02.00.00-gversion-test' 2024-09-16T06:20:11.693 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:35: run: teardown td/ver-health 2024-09-16T06:20:11.693 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-16T06:20:11.694 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T06:20:11.694 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-16T06:20:11.694 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:20:11.694 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:20:11.694 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:20:11.694 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:20:11.694 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:20:11.800 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:20:11.800 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:20:11.801 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:20:11.801 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:20:11.802 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:20:11.803 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:20:11.803 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:20:11.804 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:20:11.804 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:20:11.804 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:20:11.804 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:20:11.805 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:20:11.806 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T06:20:11.806 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-16T06:20:11.811 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:20:11.811 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:20:11.812 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:20:11.812 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.192403 2024-09-16T06:20:11.813 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:20:11.813 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:20:11.813 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-09-16T06:20:11.813 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/ver-health 0 2024-09-16T06:20:11.813 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-16T06:20:11.814 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-16T06:20:11.814 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-16T06:20:11.814 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T06:20:11.814 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T06:20:11.814 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T06:20:11.814 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T06:20:11.815 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T06:20:11.815 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T06:20:11.815 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T06:20:11.816 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T06:20:11.816 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T06:20:11.817 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T06:20:11.817 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T06:20:11.817 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T06:20:11.818 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:20:11.818 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T06:20:11.818 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T06:20:11.819 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T06:20:11.819 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:20:11.820 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-16T06:20:11.820 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-16T06:20:11.821 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T06:20:11.821 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T06:20:11.821 INFO:tasks.workunit.client.0.smithi178.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.192403 2024-09-16T06:20:11.822 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.192403 2024-09-16T06:20:11.823 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T06:20:11.823 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T06:20:11.823 INFO:tasks.workunit.client.0.smithi178.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-09-16T06:20:11.824 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T06:20:11.824 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T06:20:11.850 INFO:tasks.workunit:Stopping ['misc'] on client.0... 2024-09-16T06:20:11.850 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-16T06:20:12.177 DEBUG:teuthology.parallel:result is None 2024-09-16T06:20:12.177 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T06:20:12.202 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T06:20:12.202 DEBUG:teuthology.orchestra.run.smithi178:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-16T06:20:12.295 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T06:20:12.295 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T06:20:12.309 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T06:20:12.310 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T06:20:12.362 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T06:20:12.371 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T06:20:12.371 DEBUG:teuthology.orchestra.run.smithi178:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T06:20:12.390 INFO:teuthology.orchestra.run.smithi178.stderr:bash: line 1: ntpq: command not found 2024-09-16T06:20:12.395 INFO:teuthology.orchestra.run.smithi178.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T06:20:12.395 INFO:teuthology.orchestra.run.smithi178.stdout:=============================================================================== 2024-09-16T06:20:12.395 INFO:teuthology.orchestra.run.smithi178.stdout:^* hv01.front.sepia.ceph.com 3 7 377 27 +1212us[+1202us] +/- 50ms 2024-09-16T06:20:12.395 INFO:teuthology.orchestra.run.smithi178.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 32 -1404us[-1415us] +/- 69ms 2024-09-16T06:20:12.396 INFO:teuthology.orchestra.run.smithi178.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 25 -347us[ -347us] +/- 72ms 2024-09-16T06:20:12.396 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T06:20:12.396 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T06:20:12.405 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T06:20:12.405 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T06:20:12.444 DEBUG:teuthology.orchestra.run.smithi178:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T06:20:12.502 DEBUG:teuthology.orchestra.run.smithi178:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T06:20:12.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T06:20:12.530 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T06:20:12.540 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T06:20:12.569 INFO:teuthology.task.internal:Duration was 3943.003720 seconds 2024-09-16T06:20:12.570 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T06:20:12.611 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T06:20:12.611 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T06:20:12.650 INFO:teuthology.orchestra.run.smithi178.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T06:20:12.994 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T06:20:12.995 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi178.front.sepia.ceph.com 2024-09-16T06:20:12.995 DEBUG:teuthology.orchestra.run.smithi178:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T06:20:13.056 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T06:20:13.056 DEBUG:teuthology.orchestra.run.smithi178:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T06:20:13.123 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T06:20:13.124 DEBUG:teuthology.orchestra.run.smithi178:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T06:20:13.260 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T06:20:13.270 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T06:20:13.270 DEBUG:teuthology.orchestra.run.smithi178:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T06:20:13.295 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T06:20:13.311 DEBUG:teuthology.orchestra.run.smithi178:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:20:13.359 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern = core 2024-09-16T06:20:13.371 DEBUG:teuthology.orchestra.run.smithi178:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T06:20:13.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T06:20:13.426 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T06:20:13.435 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T06:20:13.437 DEBUG:teuthology.misc:Transferring archived files from smithi178:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7906136/remote/smithi178 2024-09-16T06:20:13.438 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T06:20:13.560 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T06:20:13.561 DEBUG:teuthology.orchestra.run.smithi178:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T06:20:13.578 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T06:20:13.589 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T06:20:13.589 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T06:20:13.627 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T06:20:13.628 DEBUG:teuthology.orchestra.run.smithi178:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T06:20:13.643 INFO:teuthology.orchestra.run.smithi178.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 06:20 /home/ubuntu/cephtest 2024-09-16T06:20:13.644 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T06:20:13.662 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T06:20:13.747 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/misc} duration: 3943.0037195682526 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-16T06:20:13.747 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T06:20:13.833 INFO:teuthology.run:pass